CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention
Sayak Dutta
Read on arXiv →
Key claim
CARVE enhances memory efficiency in recurrent models.
In plain English
Imagine you're building a system that needs to remember information over time, like a chatbot that recalls past conversations. Traditional recurrent models struggle with this because they often forget important details when they receive new information. They decide what to erase based only on the new input, which can lead to losing valuable context. This is known as memory-blind gating, and it can make the model less effective at understanding and responding accurately. When the model doesn't remember well, it can lead to poor performance in tasks that require reasoning or recalling past interactions. This is what's called the 'forgetting problem' in recurrent architectures.
The paper introduces a new method called CARVE, which addresses these issues by changing how the model decides what to erase from memory. Instead of relying solely on the new input, CARVE uses the existing memory content as a guide for what to forget. This means the model can make more informed decisions about what information is actually important to keep or discard. By reusing the output from previous computations, CARVE not only improves memory efficiency but also simplifies the architecture, making it easier to train.
In practical terms, CARVE outperforms previous models on several benchmarks, achieving better results with fewer resources. It shows a significant reduction in perplexity on WikiText and leads in common-sense reasoning tasks, all while being more efficient in terms of memory and parameters. This makes it a compelling choice for anyone looking to build systems that require effective memory management and reasoning capabilities.
The paper introduces a new approach to recurrent model architecture that addresses key limitations of existing methods.
The claims are supported by formal theorems and empirical results across multiple benchmarks.
Deep reliability assessment
The paper provides a clear architectural change (key-axis-only gating and scalar write) with theoretical guarantees for chunk-parallelism and bounded staleness, plus empirical comparisons against the best prior recurrent baseline at 1.3B params. Claims of state-of-the-art are scoped to recurrent models and RULER-style retrieval probes, and hardware parity is reported but may depend on specific kernels and training setup.
Reproducibility
No code or kernel implementation URL is provided; training details include model size (1.3B), data (100B tokens, FineWeb-Edu), hardware (H100), and three-seed averages, but reproducing the fused WY-form kernel and content-reuse path may be nontrivial without released code.
Key figure
A data-flow diagram showing q/k/v and gate projections feeding a WY-form state update where all gates act on the key axis, with a chunk-mean readout of the previous chunk’s outputs passed through zero-initialized low-rank projections to modulate a content-aware erase gate and a per-head scalar write gate.