Copy the Same, Distill the Difference: Initializing Linear Vision Transformers

Imagine you're building a computer vision model that needs to process images efficiently. You might consider using Vision Transformers (ViTs), which have become popular for their ability to handle complex visual tasks. However, a challenge arises when trying to make these models more efficient: the traditional Softmax attention mechanism can be computationally expensive, and newer linear attention methods promise to reduce this cost. The problem is that while linear ViTs are designed to be more efficient, they often struggle to match the performance of their Softmax counterparts, especially when initialized from scratch. This is what's called the initialization problem, where starting from random weights can lead to suboptimal performance.

To address this, the authors propose a method for transferring knowledge from Softmax ViTs to linear ViTs. They find that while the attention weights themselves are not directly transferable, the underlying learned representations in the MLP layers can be copied effectively. By distilling the attention behavior from Softmax models and combining it with the transferred MLP weights, linear ViTs can significantly improve their performance, even surpassing the original Softmax models in some cases. This approach not only enhances the efficiency of linear ViTs but also provides insights into how to leverage pre-trained weights across different model architectures, making it a valuable strategy for builders looking to optimize their vision models.

The work provides a new perspective on transferring weights between different attention mechanisms.

The findings are supported by consistent results across various models and datasets.