Adaptive Multi-Token Prediction: How far should we go?

Imagine you're building a language model that needs to generate text quickly and accurately, like drafting responses in a chat application. Currently, models generate one token at a time, which can be slow and inefficient, especially when the context changes. This method can lead to wasted computation if the model drafts too many tokens that are ultimately rejected, a situation known as inefficiency in token prediction. The challenge is to decide how many tokens to draft at once without knowing in advance how many will be accepted. This is where Adaptive Multi-Token Prediction (AMTP) comes in. It acts like a smart controller that learns from past experiences to determine the optimal number of tokens to draft based on the context, maximizing the speed of decoding while minimizing wasted effort. By predicting how many tokens are likely to be accepted, AMTP can adjust its strategy dynamically, leading to faster overall performance. Compared to traditional methods, AMTP can decode up to 4.1 times faster and achieves better throughput in most scenarios, making it a valuable tool for anyone looking to enhance the efficiency of language model applications.

The approach introduces a new method for optimizing token prediction in language models.

The evaluation shows significant throughput improvements across multiple settings, though specific baselines could be clearer.