EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation

Imagine you're developing a text-to-speech system that needs to convey different emotions accurately, like happiness or sadness. Current methods often struggle with this because they require extensive retraining on emotion-labeled data, which is both time-consuming and resource-intensive. When using conventional approaches, like CoCoEmo, the system treats emotion vectors as fixed directions, which can lead to a lack of precision in expressing the desired emotion. This limitation is known as poor controllability, where the output doesn't align well with the intended emotional tone.

To address this, the authors propose a new method called Emotion Residual-Enhanced Steering (EmoRES). The key insight is that emotion vectors can be broken down into two parts: a shared component that helps move speech away from a neutral tone and a residual component that fine-tunes the output toward the specific emotion requested. By controlling these components separately without needing to retrain the model, EmoRES significantly improves performance on various metrics, showing better adherence to the requested emotions and higher listener satisfaction. Compared to previous methods, this approach not only enhances the accuracy of emotional expression but also reduces the computational burden, making it a practical solution for developers in the field.

The method introduces a novel way to control emotion in TTS without retraining.

The evaluation metrics are comprehensive and show consistent improvements over existing methods.