Background
Conversational Speech Synthesis (CSS) aims to synthesize speech with human-like emotional expression and contextual consistency in user-agent interactions. However, existing CSS methods struggle to render authentic human emotions due to limited predefined emotion label spaces (e.g., seven emotion categories), while redundant multimodal tokens in multi-turn dialogue history interfere with context understanding.
AuEmoChat Framework
To address these issues, we propose AuEmoChat, a CSS framework for authentic emotion understanding and rendering.
AuEmoCodec
First, we develop AuEmoCodec, which learns a discrete authentic emotion token space from large-scale emotional speech via finite scalar quantization. This approach enables a more authentic emotion representation than limited basic emotion categories.
AuEmoToMe Algorithm
Next, we propose AuEmoToMe, an authentic-emotion-guided token merging algorithm that merges redundant tokens in multimodal dialogue history while preserving emotion-relevant context. This algorithm is integrated into an autoregressive text-speech model to predict the target authentic emotion token and speech tokens.
Authentic Emotion Flow Matching
Finally, we propose Authentic Emotion Flow Matching, which renders speech by jointly conditioning on merged dialogue context, target authentic emotion, and acoustic priors.
Experimental Results
Extensive experiments on the NCSSD-EmCap dataset demonstrate that AuEmoChat outperforms state-of-the-art CSS baselines and generates more expressive and authentic emotional speech.
Blogger's Review: AuEmoChat significantly enhances the emotional expressiveness of conversational speech synthesis by introducing authentic emotion tokens and an efficient context merging algorithm, paving the way for new possibilities in human-computer interaction and its potential applications are worth watching closely.