Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

Imagine you're developing a text-to-speech (TTS) system that needs to sound as natural as possible. Currently, most evaluations rely on Mean Opinion Scores (MOS) that focus heavily on overall sound quality, but this approach often misses the subtleties of what makes speech feel natural to listeners. For instance, it might overlook issues like intonation, rhythm, or emotional expression, which are crucial for a realistic experience. This limitation is what's called a collapse onto acoustic signal quality, meaning it doesn't capture the full spectrum of human perception in speech. Additionally, newer methods using Audio Large Language Models (Audio-LLMs) show promise but struggle with consistency across different speech dimensions, leading to selective and prompt-dependent evaluations. This is where the new approach comes in. By breaking down 'naturalness' into ten distinct perceptual dimensions, the authors created a comprehensive evaluation benchmark that includes 860 utterances rated by trained linguists. This allows for a more nuanced understanding of TTS quality, revealing that existing methods often fail to address the breadth of linguistically structured speech errors. With this new dataset and evaluation framework, builders can better assess and improve TTS systems, leading to more natural-sounding speech outputs.

The paper introduces a new evaluation framework for TTS that dissects naturalness into distinct dimensions.

The benchmarking results are based on a well-annotated dataset and rigorous evaluation of existing methods.