Data-efficient Targeted Token-level Preference Optimization for LLM-based Text-to-Speech
This addresses the data efficiency and fine-grained alignment challenges in TTS systems, particularly for languages like Japanese, representing a novel method for a known bottleneck.
The paper tackles the problem of limited paired data and coarse optimization in text-to-speech systems by proposing TKTO, a method that eliminates the need for paired data and enables token-level preference optimization. The result is a 39% improvement in Japanese TTS accuracy and a 54% reduction in CER, with 12.8 times stronger reward for targeted tokens.
Aligning text-to-speech (TTS) system outputs with human feedback through preference optimization has been shown to effectively improve the robustness and naturalness of language model-based TTS models. Current approaches primarily require paired desirable and undesirable samples at the utterance level. However, such pairs are often limited in TTS output data, and utterance-level formulation prevents fine-grained token-level optimization needed for accurate pronunciation alignment. In this study, we propose TKTO that eliminates the need for paired data, enabling a more data-efficient training paradigm, and directly targets token-level units, automatically providing fine-grained alignment signals without token-level annotations. TKTO improves the challenging Japanese TTS accuracy by 39% and reduces CER by 54%, automatically assigning 12.8 times stronger reward to targeted tokens.