DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees

Imagine you're building a language model that needs to generate text quickly and accurately, like drafting responses in a chat application. Current methods often rely on generating tokens one at a time, which can be slow and inefficient, especially when trying to verify multiple options simultaneously. This sequential approach can lead to bottlenecks, where the model struggles to keep up with user expectations, resulting in delays or less coherent outputs. This is what's called autoregressive decoding, and while it works, it can be limiting in terms of speed and flexibility.

To address these issues, DARTree proposes a new way to handle speculative decoding by organizing token proposals into a tree structure rather than a single chain. This allows the model to evaluate multiple potential token sequences in parallel, significantly speeding up the process. By constructing a fixed-width candidate tree and applying best-first pruning, DARTree can efficiently select the most promising paths for verification without the overhead of traditional sequential methods. Compared to previous approaches, DARTree achieves a notable increase in both the number of tokens accepted per round and overall decoding speed, making it a compelling option for developers looking to enhance the performance of autoregressive models in real-world applications.

DARTree introduces a novel approach to speculative decoding by extending correction from chains to trees.

The method shows solid performance across multiple benchmarks, though it lacks extensive baseline comparisons.