Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning

Imagine you're building a language model that needs to solve complex problems by reasoning through long pieces of text. The challenge is that these models often fall into a trap where they just copy parts of the input instead of generating original solutions. This behavior, known as repetitive copying, becomes more pronounced as the context length increases, leading to incorrect answers when the model fails to focus on the relevant information. Essentially, the models struggle with distinguishing between key evidence that helps solve the problem and irrelevant details that distract them. To tackle this issue, the authors propose a method called GEAR, which enhances the model's training by rewarding it for focusing on important evidence while penalizing it for copying irrelevant context. They also created a system to automatically generate training data that highlights this key evidence. By applying GEAR, they observed consistent improvements in model performance, particularly in longer contexts, reducing the tendency to copy and increasing the accuracy of the answers. This approach suggests that improving how models ground their reasoning in relevant evidence is crucial for advancing their capabilities in complex tasks.

The introduction of GEAR addresses a significant failure mode in long-context LLMs.

The validation across multiple model scales and benchmarks supports the findings.