Co-LMLM: Continuous-Query Limited Memory Language Models
Yair Feldman, Linxi Zhao, Nathan Godey, Dongyoung Go, Yilun Hua, Kilian Q. Weinberger, Jennifer J. Sun, Yoav Artzi
Read on arXiv →
Key claim
CO-LMLM outperforms traditional models in factual accuracy.
In plain English
Imagine trying to build a language model that can answer questions accurately without just memorizing facts. Traditional models often struggle because they store knowledge in their weights, which can lead to outdated or incorrect information. When they encounter a question, they might not have the right answer readily available, which is a problem known as knowledge retention failure. This can be especially problematic when the model is asked about recent events or niche topics that weren't part of its training data.
To address this, the paper introduces a new approach where the model doesn't just memorize facts but instead pulls information from an external knowledge base (KB) as needed. This is called continuous-query LMLM. The model generates flexible vector queries to retrieve relevant information from the KB, which allows it to provide more accurate and up-to-date answers. This method also integrates human-readable knowledge into its responses, making it easier to verify the information provided.
What sets CO-LMLM apart from previous models is its ability to use continuous keys paired with textual knowledge values, rather than relying on traditional relational databases. This flexibility means that the model can access a broader range of information beyond just what's available in structured formats like Wikipedia. In practical terms, this leads to lower perplexity and higher factual precision, even outperforming models trained on much larger datasets. For anyone building applications that require accurate and timely information retrieval, this approach offers a promising solution.
The introduction of continuous-query LMLMs represents a significant shift in how knowledge is accessed and utilized in language models.
The claims are supported by strong experimental results across multiple datasets and model scales.
Deep reliability assessment
The methodology supports the claim that CO-LMLM improves perplexity and factual precision over prior LMLMs and vanilla LLMs, but the reliance on specific datasets like Wikipedia and FineWeb-Edu may limit generalizability. The paper's claims about scalability and expressivity are supported by experiments but may not fully account for real-world variability in knowledge retrieval tasks.
Reproducibility
No open source code or dataset URL is mentioned in the paper.
Key figure
Figure 1 illustrates the knowledge separation across three regimes: a standard LLM with RAG, REL-LMLM with a relational KB, and CO-LMLM with a continuous-query knowledge base.