Differentiable Reward Optimization for LLM based TTS system
This addresses the challenge of enhancing pronunciation accuracy and controllability in TTS systems for applications requiring high-quality speech synthesis, representing a novel method for a known bottleneck rather than a foundational breakthrough.
The paper tackled the problem of improving text-to-speech systems by proposing Differentiable Reward Optimization (DiffRO), which directly computes rewards from neural codec tokens and uses Gumbel-Softmax for differentiability, resulting in state-of-the-art WER results on the seed-tts-eval benchmark and enabling zero-shot control of emotional and quality attributes.
This paper proposes a novel Differentiable Reward Optimization (DiffRO) method aimed at enhancing the performance of neural codec language models based text-to-speech (TTS) systems. In contrast to conventional reinforcement learning from human feedback (RLHF) approaches applied to TTS, DiffRO directly compute the rewards based on neural codec tokens, rather than relying on synthesized audio. Furthermore, we employ the Gumbel-Softmax technique to render the reward function differentiable, thereby streamlining the RLHF training process. Additionally, we introduce a multi-task reward (MTR) model which can provide feedback from different perspectives and find that it can augment the system's capability to follow instructions effectively.Experimental results indicate that DiffRO significantly improves the pronunciation accuracy of the TTS system, achieving state-of-the-art (SOTA) WER results on the seed-tts-eval benchmark. Moreover, with the integration of the MTR model, we demonstrate the ability to control emotional and quality attributes in a zero-shot manner.