DEX-TTS: Diffusion-based EXpressive Text-to-Speech with Style Modeling on Time Variability
This work addresses limitations in expressive TTS for applications like virtual assistants or audiobooks, but it is incremental as it builds on existing diffusion TTS frameworks with novel style modeling.
The paper tackles the problem of generating expressive speech from text using reference audio by proposing DEX-TTS, a diffusion-based model that separates styles into time-invariant and time-variant categories, achieving outstanding performance in objective and subjective evaluations on multi-speaker and emotional datasets without pre-training.
Expressive Text-to-Speech (TTS) using reference speech has been studied extensively to synthesize natural speech, but there are limitations to obtaining well-represented styles and improving model generalization ability. In this study, we present Diffusion-based EXpressive TTS (DEX-TTS), an acoustic model designed for reference-based speech synthesis with enhanced style representations. Based on a general diffusion TTS framework, DEX-TTS includes encoders and adapters to handle styles extracted from reference speech. Key innovations contain the differentiation of styles into time-invariant and time-variant categories for effective style extraction, as well as the design of encoders and adapters with high generalization ability. In addition, we introduce overlapping patchify and convolution-frequency patch embedding strategies to improve DiT-based diffusion networks for TTS. DEX-TTS yields outstanding performance in terms of objective and subjective evaluation in English multi-speaker and emotional multi-speaker datasets, without relying on pre-training strategies. Lastly, the comparison results for the general TTS on a single-speaker dataset verify the effectiveness of our enhanced diffusion backbone. Demos are available here.