The Sirens' Song: When Proximal Background Context Overshadows Distant Evidence

Imagine you're building a long-context language model that needs to pull in relevant information from a vast amount of text. The challenge isn't just about how far away that information is; it's often about how much irrelevant information is competing for attention right next to it. This issue, known as the Proximity Trap, can lead to models missing out on valuable distant evidence because they get overwhelmed by nearby, less relevant details. Current approaches mainly focus on overcoming the distance, but they don't adequately address this competition from proximal context. To tackle this, LYRA introduces a method that reshapes how context is retrieved, directing more attention to the most relevant information while still keeping track of where everything is located in the text. This is achieved through a t-distributed directional matching mechanism that aligns relevance more effectively. The authors also present ProxBench, a new benchmark designed to evaluate how well models utilize distant evidence in the presence of distracting nearby information. Compared to previous work, LYRA not only improves the retrieval of relevant information over longer contexts but also provides a more nuanced way to assess model performance in real-world scenarios where irrelevant context can interfere. For builders, this means better tools for developing models that can effectively sift through large amounts of data and focus on what truly matters.

Introduces a new mechanism to improve long-context retrieval by addressing the Proximity Trap.

Extensive experiments on multiple benchmarks support the claims of improved performance.