Diffusion-GR2: Diffusion Generative Reasoning Re-ranker
Zhuoxuan Zhang, Kangqi Ni, Yuhang Chen, Mingfu Liang, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Frank Shyu, Adam, Song, Sandeep Pandey, Luke Simon, Tianlong Chen, Xi Liu
Read on arXiv →
Key claim
Diffusion-GR2 achieves 2.4-3.5x faster decoding with high accuracy.
In plain English
Imagine you're building a recommendation system that needs to quickly sort through a list of options to find the best ones. Traditional methods often rely on a sequential process where each reasoning step takes time, leading to slow performance. This is especially problematic when you have a lot of candidates to evaluate, as the time it takes to generate a recommendation can become a bottleneck. This is what's called slow inference in autoregressive models, where each reasoning token is processed one after the other, making it inefficient for real-time applications.
To address this, some researchers have tried to adapt these models to work in parallel, which can speed things up significantly. However, this approach can lead to two main issues. First, when reasoning steps are processed simultaneously, the model might produce invalid rankings, such as duplicates or missing options. This is known as the structural gap. Second, if the model is trained on fixed examples that don't match its own way of generating outputs during inference, it can lead to inaccuracies, referred to as the distributional gap.
The authors of this paper propose a solution called Diffusion-GR2, which effectively converts an autoregressive reasoning model into a block-diffusion model. They introduce a process called conversion fine-tuning that helps the model learn to generate valid rankings on its own, without relying on external constraints. Then, they use on-policy distillation to train the model on its own outputs, ensuring it learns from its own reasoning process. Finally, they apply reinforcement learning to fine-tune the model further based on a reward system for better rankings. The result is a model that not only recovers accuracy close to the original autoregressive model but also significantly increases the speed of generating recommendations, making it much more practical for real-world applications.
The paper introduces a new method for improving the efficiency and accuracy of generative reasoning re-rankers, which is a significant extension of existing techniques.
The claims are well-supported by experiments demonstrating recovery of accuracy and increased throughput, with appropriate ablations.
Deep reliability assessment
The methodology supports the claim that Diffusion-GR2 can achieve near-parity with AR re-rankers while significantly increasing throughput. However, the claim of closing accuracy gaps may be overclaimed without more extensive testing across diverse datasets.
Reproducibility
No open source code or dataset is mentioned in the paper.
Key figure
Figure 1 provides an overview of the Diffusion-GR2 system, illustrating the conversion process from an AR GR2 re-ranker to a block-diffusion re-ranker using conversion fine-tuning, on-policy distillation, and reinforcement learning.