Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance
This work addresses controllability issues in text-to-speech generation for applications requiring high-quality, reliable speech synthesis, representing an incremental improvement with specific optimizations.
The paper tackles the problem of lack of controllability in autoregressive speech token generation models, which leads to hallucinations and undesired vocalizations, by introducing Koel-TTS, a suite of enhanced encoder-decoder Transformer TTS models that incorporate preference alignment and classifier-free guidance, resulting in significant improvements in speaker similarity, intelligibility, and naturalness, outperforming state-of-the-art models on a smaller dataset.
While autoregressive speech token generation models produce speech with remarkable variety and naturalness, their inherent lack of controllability often results in issues such as hallucinations and undesired vocalizations that do not conform to conditioning inputs. We introduce Koel-TTS, a suite of enhanced encoder-decoder Transformer TTS models that address these challenges by incorporating preference alignment techniques guided by automatic speech recognition and speaker verification models. Additionally, we incorporate classifier-free guidance to further improve synthesis adherence to the transcript and reference speaker audio. Our experiments demonstrate that these optimizations significantly enhance target speaker similarity, intelligibility, and naturalness of synthesized speech. Notably, Koel-TTS directly maps text and context audio to acoustic tokens, and on the aforementioned metrics, outperforms state-of-the-art TTS models, despite being trained on a significantly smaller dataset. Audio samples and demos are available on our website.