RetroThinker: Enabling Retrospective Thinking in Speech LLMs
Yi-Jen Shih, Puyuan Peng, Abdelrahman Mohamed, David Harwath
Read on arXiv →
Key claim
RetroThinker improves accuracy-latency trade-off in SpeechLLMs.
In plain English
Imagine you're building a voice assistant that needs to understand and respond to users in real-time, like during a conversation. Current systems often struggle with complex reasoning tasks, especially when they have to process speech and text simultaneously. This leads to issues like misinterpretation of user intent or delayed responses, which can frustrate users. This is what's called the accuracy-latency trade-off, where improving one often worsens the other. To address this, a new approach called RetroThinker has been developed, which allows a speech model to revise its reasoning as it listens and responds. By combining supervised fine-tuning with a method that optimizes for preference based on the length of reasoning, RetroThinker can correct its thought process on the fly. This means that while the user is speaking, the model can adjust its responses to be more accurate without significantly increasing latency. Compared to previous models, RetroThinker achieves an 11% absolute accuracy gain while maintaining similar response times, making it a promising option for real-time applications.
The approach of dynamically revising reasoning during inference is a meaningful extension to existing models.
The evaluation on the GSM8K benchmark shows solid improvements over non-retrospective baselines.
Deep reliability assessment
The methodology supports improved accuracy-latency trade-offs in SpeechLLMs through retrospective thinking, but the extent of improvement may be overclaimed without broader benchmarks.
Reproducibility
No open source code or dataset is mentioned in the paper.
Key figure
Figure 1 illustrates the RetroThinker framework, combining early reasoning and retrospective thinking to improve accuracy-latency trade-offs.