CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering
This addresses the limitation of single-emotion TTS for applications requiring human-like expressive speech, though it is incremental as it builds on existing steering methods.
The paper tackled the problem of generating nuanced, mixed, or text-mismatched emotional speech in text-to-speech systems by introducing a systematic activation steering framework, demonstrating that emotional prosody is primarily synthesized by the language module and enabling composable emotional control with natural results.
Emotional expression in human speech is nuanced and compositional, often involving multiple, sometimes conflicting, affective cues that may diverge from linguistic content. In contrast, most expressive text-to-speech systems enforce a single utterance-level emotion, collapsing affective diversity and suppressing mixed or text-emotion-misaligned expression. While activation steering via latent direction vectors offers a promising solution, it remains unclear whether emotion representations are linearly steerable in TTS, where steering should be applied within hybrid TTS architectures, and how such complex emotion behaviors should be evaluated. This paper presents the first systematic analysis of activation steering for emotional control in hybrid TTS models, introducing a quantitative, controllable steering framework, and multi-rater evaluation protocols that enable composable mixed-emotion synthesis and reliable text-emotion mismatch synthesis. Our results demonstrate, for the first time, that emotional prosody and expressive variability are primarily synthesized by the TTS language module instead of the flow-matching module, and also provide a lightweight steering approach for generating natural, human-like emotional speech.