How to Loop MoE: Flatten the Experts, Untie the Attention

Imagine you're building a large language model that needs to efficiently use its resources while maintaining high performance. Current models often struggle with balancing the load across multiple experts, leading to underutilization of their capabilities. This is particularly evident in sparse mixture-of-experts (MoE) models, where only a few experts are activated for each input, which can limit their effectiveness. The challenge is to find a way to make better use of these experts without increasing the overall complexity of the model. This is what's called the load balancing problem. To address this, the authors propose a method called Foil, which combines the principles of looped transformers and MoE. The intuition behind Foil is to flatten the expert layers, allowing for more experts to be considered in each routing decision while maintaining a fixed number of parameters and compute. By untying the attention parameters for each pass, Foil enables more tailored routing decisions, leading to improved performance. The results show that Foil outperforms traditional looped models, achieving lower pretraining loss and better downstream accuracy, suggesting that a more flexible approach to expert usage can yield significant benefits. This means that for builders, adopting such a method could lead to more efficient models that leverage their resources more effectively.

The approach of combining looped transformers with MoE is a meaningful extension of existing architectures.

The experiments show consistent improvements over baselines, though more details on the methodology would strengthen the claims.