SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models

Imagine you're building a system that can take spoken questions and respond with synthetic speech. You want it to sound natural, like a real conversation, but current benchmarks mainly focus on how accurately the system understands and generates speech. This is where things can go wrong: while a model might produce clear and correct answers, it could still feel robotic or awkward in a back-and-forth dialogue. For instance, it might interrupt too often, take too long to respond, or fail to adapt its tone to the emotional context of the conversation. These issues are what's called naturalness failures in conversational AI.

To address these shortcomings, the authors created SPEARBench, a new benchmark specifically designed to evaluate how naturally speech-to-speech models interact in conversations. Instead of just measuring accuracy, SPEARBench looks at various factors like response timing, emotional tone, and how well the model maintains consistency in language and dialect. By using controlled dialogue prompts and comparing model outputs to human responses, they provide a more comprehensive view of conversational quality.

What sets this work apart from previous benchmarks is its multidimensional approach to evaluation. It shows that even when models achieve high technical performance, they can still fall short in mimicking human conversational behavior. For anyone building conversational systems, this means you need to focus not just on getting the right answers, but also on how those answers are delivered to ensure a more human-like interaction.

The introduction of SPEARBench provides a new framework for evaluating naturalness in speech-to-speech models, extending existing benchmarks.

The evaluation includes multiple models and dimensions, but may lack extensive baselines for all claims.