STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Imagine you're building a large language model that needs to serve many users simultaneously. As these models grow, they require significant memory to maintain their performance, especially when using linear attention mechanisms that rely on recurrent states. However, these states can become a memory bottleneck, particularly when serving multiple requests at once. A common approach to mitigate this is to quantize these states to lower precision, but this often leads to accuracy issues due to the propagation of quantization errors over time and across different parts of the model. This is what's called quantization error propagation, which can severely degrade performance in practical applications. To tackle this, the authors propose STEPQuant, a framework that intelligently allocates quantization precision based on how long the memory is expected to last and the impact of different parts of the model on the final output. By analyzing the distribution of state magnitudes and their effects on errors, STEPQuant can maintain high accuracy even with reduced precision. In experiments with large models like Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct, STEPQuant achieves impressive results, closely matching full precision accuracy while significantly reducing memory usage. This means that for builders, integrating STEPQuant could lead to more efficient models that are easier to deploy in real-world applications without sacrificing performance.

Introduces a new quantization framework that addresses memory bottlenecks in linear attention models.

Demonstrates effectiveness through experiments on large models, but lacks extensive baseline comparisons.