It turns out that there might be an elegant way around bloated context windows and slow inference speeds. Instead of stuffing thousands of text tokens into a prompt like standard retrieval setups do, Memorilla approach encodes an entire document collection into just 16 query dependent latent tokens. The system drops these soft memory tokens directly into the input embedding space of a frozen LLM. Doing so averages 394 times fewer document tokens than a full context baseline and 113 times fewer than a standard fifty document retrieval method. Aside from saving massive amounts of compute overhead, the system also beats full context inference on complex multi document reasoning tasks. It hit 84.33 percent accuracy on the FactKG dataset compared to 80.43 percent for full context while using 108 times fewer tokens to represent the source documents.
This approach also has implications for continuous learning and agentic workflows. Having persistent memory prevents the agent from getting stuck in stupid navigation loops between previously explored items. Because the initial document encoding is a one time shared cost, the decoder only processes a small number of memory tokens. You get continuous adaptation to new environments without overwriting prior knowledge or blowing up your active context limit. It's a really clever way to decouple persistent knowledge storage from the active context window.