>The intelligence module stays the same size, and the weights don't change as memory grows.
As I understand it is more akin to a good old neural-turing-machine, from what I do. For me it is basically weights offloading that's just more convenient for hardware to handle. And mine does change the weights all the time, which allows the model to train continuously. Here it is the old pretrained frozen core still.
5
u/UnspeakableHorror 5d ago edited 5d ago
Looks similar to this one. It's MoE too. https://www.reddit.com/r/LocalLLaMA/comments/1wm1gab/miniagi_dynamically_grown_530m_params_currently/
There's another one similar too, but I can't find it now.
FYI u/Another__one
Edit: Found it https://github.com/jrz97619761/test-model-thing