Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA

Imagine you're building a machine learning model that needs to handle a wide variety of tasks, from understanding language to solving math problems. In many cases, these models use a fixed number of experts to process information, which can lead to inefficiencies. For instance, if the model is overly confident about easy tokens, it might waste resources on them while neglecting harder tokens that need more attention. This mismatch can result in suboptimal performance, especially when the model encounters ambiguous or complex inputs — a situation known as over- or under-utilization of resources.

To address this, the authors propose a method called CARE, which allows the model to activate experts based on the uncertainty of each token. Instead of sticking to a rigid number of experts, CARE uses a dynamic approach where it activates experts in a way that reflects the model's confidence in its predictions. This means that for tokens the model is less sure about, it can bring in more experts to help, while still being efficient with resources. The results show that CARE not only matches the performance of fixed expert systems but does so while using fewer resources, making it a more efficient choice for builders looking to optimize their models.

CARE introduces a novel approach to expert activation based on per-token uncertainty.

The evaluation across multiple benchmarks shows solid improvements over existing methods.