UniSS: Unified Expressive Speech-to-Speech Translation with Your Voice
This work addresses a key challenge in speech-to-speech translation for applications requiring personalized and emotional communication, though it appears incremental as it builds on existing text-based LLM frameworks.
The paper tackles the problem of expressive speech-to-speech translation by preserving speaker identity and emotional style, introducing UniSS, a single-stage framework that integrates with large language models and uses a cross-modal chain-of-thought prompting process, resulting in significant outperformance over previous methods in translation fidelity and speech quality.
The ultimate goal of expressive speech-to-speech translation (S2ST) is to accurately translate spoken content while preserving the speaker identity and emotional style. However, progress in this field is largely hindered by three key challenges: the scarcity of paired speech data that retains expressive styles, the complexity of multi-stage processing pipelines, and the limited transfer of translation capabilities from large language models (LLMs). In this work, we address these challenges by introducing UniSS, a novel single-stage framework for expressive S2ST. Our approach features carefully designed speech semantic and style modeling, enabling seamless integration with existing text-based LLM frameworks to develop a unified text-speech language model. To transfer translation capabilities from text to speech, we propose a cross-modal chain-of-thought prompting process that progressively aligns audio semantics with text and ensures style preservation in the decoded results. Furthermore, we construct and release a large-scale, high-quality expressive S2ST dataset, UniST, comprising 44.8k hours of data. Experimental results show that UniSS significantly outperforms previous methods in translation fidelity and speech quality while preserving voice, emotion, and duration consistency. Our work establishes a simpler and more effective paradigm for building the next generation of expressive S2ST systems. Audio samples are available at https://cmots.github.io/uniss-demo.