FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

Imagine you're working with large language models that need to handle long inputs, like lengthy documents or complex queries. These models often rely on memory caches to store information from previous layers, but as the context length increases, the memory and bandwidth costs can skyrocket. This can lead to inefficiencies, especially when trying to maintain accuracy in tasks like question answering or summarization. When you try to compress these caches aggressively, you might lose important details that are crucial for understanding and reasoning — this is what's called the loss of layer-specific evidence.

To tackle this issue, the authors propose a new method called FreqDepthKV. Instead of treating all cache data the same, this approach breaks down the information into two parts: shared low-frequency components that can be reused across layers and sparse high-frequency residuals that capture the unique details. A smart online probe then decides how to use these components based on their importance for the task at hand, allowing the model to adapt to different types of prompts without needing to retrain.

What’s exciting is that FreqDepthKV not only reduces the memory footprint significantly but also maintains high accuracy across various tasks. For instance, with a 32k-token input, it achieves performance metrics that closely match those of full KV caches while being much more efficient. This means that for builders, using this method can lead to faster processing times and lower resource costs, making it a practical solution for deploying long-context models in real-world applications.

The method introduces a novel approach to cache compression that adapts to prompt structure, significantly improving efficiency.

The results are supported by multiple benchmarks and demonstrate clear improvements over prior methods.