How Local Mixing Encodes Relative Position in Global NoPE Attention

Imagine you're building a language model that needs to understand the order of words in a sentence. Traditional transformer models use explicit position encodings to help with this, but they can struggle with longer sequences or when the context shifts. This is where methods like rotary position encoding (RoPE) come in, but they often assume that explicit encodings are necessary for success. However, recent approaches have shown that you can achieve good results without these encodings by using techniques like sliding window attention (SWA) and gated linear attention, which don't explicitly encode position in their global attention layers. This can lead to issues, such as losing track of the sequence order over long texts, which is known as the recency bias problem.

The authors propose that hybrid models, which combine local mixing layers with global attention, can implicitly maintain positional information through a recency bias in the residual stream. This means that even without explicit position encodings, these models can still effectively manage long sequences by leveraging the way they process information. By understanding how this recency bias works, builders can gain insights into designing models that can handle longer sequences more effectively, potentially leading to better performance in real-world applications.

The paper provides a new understanding of implicit position encoding in hybrid attention models.

The findings are supported by both theoretical and empirical evidence, though the institution is unknown.