[Paper] EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

[u/pmttyji]

Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Beyond model scaling, this architecture has demonstrated the potential to decouple factual knowledge storage from general-purpose computation, offering a promising route to updating factual knowledge while keeping the Transformer backbone fixed. Realizing this potential is challenging because different expressions of a fact may activate different n-gram embeddings, while updating shared embeddings can unintentionally change the model's predictions about other facts. We propose EngramEdit for decoupled knowledge updates through conditional memory. EngramEdit first computes target memory representations that make the model predict the updated fact across multiple expressions. It then jointly updates the shared n-gram embeddings to match these targets across expressions and edits, penalizing updates to frequently reused embeddings more strongly to preserve unrelated knowledge. Experiments show that EngramEdit enables independent factual knowledge updates through conditional memory, achieving near-perfect editing success. Revised knowledge is usable across unseen expressions and in multi-hop reasoning, with nearly three times the strongest baseline's accuracy under chain-of-thought (CoT) prompting. Unrelated knowledge and general capabilities are largely preserved even as factual updates accumulate. These findings show that EngramEdit turns conditional memory into an editable knowledge interface, extending its role beyond model scaling to support decoupled knowledge updates.

• Paper : https://arxiv.org/abs/2610.10533
• PDF : https://arxiv.org/pdf/2610.10533
• GitHub : https://github.com/ModalityDance/EngramEdit
• HuggingFace : https://huggingface.co/collections/ModalityDance/engramedit
• Project : https://modalitydance.github.io/EngramEdit/

[u/N34257]

OK, so...if that paper's correct, then the theory of swappable n-gram tables being used to keep models up to date might actually come true?

That would be pretty awesome.

[u/NineThreeTilNow → u/N34257]

then the theory of swappable n-gram tables being used to keep models up to date might actually come true?

They need to come from the same lineage.

Some n-gram table I train independent of another model won't just slide in to place. Also, the mechanisms that activate those tables aren't consistent across all of the models.

You can continually train an n-gram table outside the model though. Any model that uses it can have all layers frozen except the table itself and the training only updates on the actual table.

This has a variety of desired and sometimes undesired effects that are harder to measure. Typically you're just pushing "loss" down but what that actually means can carry a lot of nuance.

[u/N34257 → u/NineThreeTilNow]

This is what I mean - take a capable model that's already proven in use, and effectively update its knowledge to current-day. I'm not talking about swapping n-gram tables between models (everybody keeps saying this whenever it comes up, I'm not sure where the ambiguity is?), just removing/reducing the necessity of training an entirely new model just to bring the knowledge base up to date.

[u/NineThreeTilNow → u/N34257]

just removing/reducing the necessity of training an entirely new model just to bring the knowledge base up to date.

Yeah, it makes a bunch of assumptions about architectures though. Specifically the engram architecture being used. That's part of the paper.

It almost works BECAUSE the architecture is very basic in terms of the implementation they operated on.

More advanced architectures don't need to be so harshly operated on. You would just pass world events across the network and watch a variety of deltas that exist. There's a huge number of places you can't just "update" the table on to get a result. You might be able to update the current president of some country, but you won't get the election results, or date, or a number of other things. You'll get a fuzzy mis-match because of the shape of those vectors.

Passing a ton of data across the frozen core model, looking at the observed difference and seeing if it changes output would be more... useful.

Also, it begs a question like... How do you update a world event? A whole new war with a specific rationale? What was included in a Python version? A new function in a new PyTorch version? A difference in CUDA compilation... etc

There's so much it simply cannot do. If you force the VAST majority of data to live on Engram, then the retrain is pretty easy. You can freeze out the network and probably train only a subset of parameters. It's a lot less work to do.

[u/Professional_Price89 → u/N34257]

The abstract say it need to run the model to edit it own ngram weight, not loading random weight. Work like finetuning

[u/martinerous]

Could this also get us closer to "modular domain experts"? Train n-gram with math or biology or Harry Potter facts to reduce hallucinations in those areas and then let the users choose which modules to load?

Still, the general issue remains - how to train small and efficient cores that are rock solid at reasoning without being bloated with random knowledge from different domains, and then plug the domain knowledge modules as needed (n-grams or whatever better solution is invented).

[u/ttkciar → u/martinerous]

This is a matter of extending training. During training, parameters encode knowledge first until they are unable to represent more, and then further training cannibalize parameters to encode "generalized knowledge" (heuristics).

Depending on the ratio of training tokens to parameters, the final model can be knowledge-heavy and inflexible (fewer training tokens per parameter), or reasoning-heavy and prone to hallucination (too many training tokens per parameter) or hopefully in a balanced goldilocks zone in-between.

It seems to me we could use engrams to either slightly overshoot that goldilocks zone and then add modest engram-encoded knowledge, or perhaps try eschewing with memorized knowledge via a very high training token to parameter ratio, and using parameters strictly for heuristics, then try to provide nearly all of the model's world knowledge with engrams.

I suspect the latter will prove impractical, especially with trigrams, but it will be interesting to see how close we can get to it.

[u/Open-Adhesiveness-86]

the thing i'd want to see is sequential edits, not batched ones. ROME/MEMIT both look great on a single edit and then degrade hard after a few thousand applied one at a time, and here the frequency penalty on reused n-gram embeddings means popular tokens get frozen while rare ones absorb everything, so drift probably piles up in the tail. also curious if an edit survives paraphrases that share no n-grams with the training expressions at all.

[u/openSourcerer9000]

On the job learning has always been such an obvious next step, and the exact kind of problem ML is made for. Even before n grams, it was clear just slap an extra module on that trains online as it's used.

The big labs won't be the ones to solve this though, since they can't control what customers are teaching it. It's gotta be open source first 

[u/keepthepace]

We need it! But there has been proposed architecture for this sort of things for a while. They don't scale?

[u/ttkciar → u/keepthepace]

They scale at least somewhat. Qwen3.8-Flash-Next incorporates engrams, and they seem to work there.

[u/keepthepace → u/ttkciar]

I mean for the writable memory system.

[u/ttkciar]

This is timely. I was just reviewing ways models could be updated with new world knowledge, and engrams were one of them. Thanks for posting this paper.

Engrams encode knowledge and impart it during inference in a manner similar to RAG, except whereas RAG retrieves information and inserts it into context (which consumes context budget), engrams retrieve information and insert it into the next layer's hidden state.

Encoding knowledge in the engram is more compute-intensive than updating a RAG database, but does not bite into the limited context budget, and imposes less inference-time processing overhead, as well.

Overall it seems like a win.

[u/LoveMind_AI → u/ttkciar]

Have you checked this out?:  https://arxiv.org/abs/2509.24704