Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection

Imagine you're building a language model that needs to generate responses based on a large amount of context, like a chatbot that pulls information from a vast database. Currently, systems like retrieval-augmented generation (RAG) face a significant challenge: as the context length increases, the time and resources needed to process that information grow, leading to slow response times. This is particularly problematic when using Transformer models, which require a growing cache of information that can become unwieldy. This situation is known as the KV-cache problem, where the model's efficiency drops as it tries to manage more data.

To address these issues, the authors propose a new method called PRECOG, which leverages the unique properties of State-Space Models (SSMs). Instead of processing context in real-time, PRECOG pre-encodes the necessary information into fixed-size hidden states that can be injected directly during query time. This means that instead of waiting for the model to digest long contexts, it can retrieve and utilize a summary instantly, drastically reducing the time it takes to generate responses. The results are impressive, with a speedup from around 27 seconds to less than 6 milliseconds, making the system interactive and practical for real-world applications. This method is not feasible with traditional Transformer architectures, marking a significant shift in how we can approach context management in language models.

The approach introduces a fundamentally new way to handle context in language models.

The results show significant performance improvements with clear metrics and comparisons.