ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning
Yanjun Zhao, Ruizhong Qiu, Tianxin Wei, Yuanchen Bei, Zhining Liu, Lingjie Chen, Ismini Lourentzou, Hanghang Tong, Jingrui He
Read on arXiv →
Key claim
RECONTEXT improves long-context reasoning without retraining models.
In plain English
Imagine you're using a large language model to answer questions based on long documents. The challenge is that while these models can read a lot of text, they often miss important details that are already in front of them. This can lead to answers that are incomplete or not well-supported by the evidence. This issue is known as ineffective context utilization, where the model has access to relevant information but fails to leverage it properly.
Currently, many models try to handle long contexts by simply increasing the amount of text they can process. However, this doesn't solve the problem of how to organize and retrieve the most relevant pieces of information when generating an answer. As a result, models can still struggle to provide accurate responses, especially when the context is complex or lengthy. This is what's called the gap between context access and effective context utilization.
The approach introduced in this paper, called Recursive Evidence Replay, addresses these failures by creating a system that organizes relevant evidence from the input text and replays it during the answer generation process. Instead of needing to retrain the model or add external memory, RECONTEXT uses internal signals to identify and prioritize the most relevant information. This means that the model can focus on the best evidence available without losing sight of the full context.
In practical terms, this method has shown to significantly improve how well models utilize evidence from long contexts across various datasets. By achieving the best average rank on multiple model backbones, RECONTEXT offers a promising solution for anyone looking to deploy large language models in applications where understanding and reasoning over extensive information is crucial.
RECONTEXT introduces a novel method for improving long-context reasoning without requiring training.
The paper provides solid experimental validation across multiple datasets and models, supporting its claims.
Deep reliability assessment
The methodology supports improved evidence utilization in long-context reasoning without training or external memory, but the claims of consistent improvement across models may be overclaimed without more diverse benchmarks.
Reproducibility
Yes, the paper provides open source code at the mentioned GitHub repository.
Key figure
Figure 1 shows that the top 0.1% of context tokens account for about 50% to 80% of the accumulated relevance score across three LLMs.