Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets
This addresses hallucinations in TTS for users needing accurate speech synthesis, representing a strong specific gain rather than a broad breakthrough.
The paper tackles the problem of hallucinated speech in language model-based text-to-speech systems by proposing a post-training framework called GOAT, which reduces character error rates by over 50% and lowers uncertainty by up to 58% on challenging test cases.
Language Model (LM)-based Text-to-Speech (TTS) systems often generate hallucinated speech that deviates from input text. Existing mitigation strategies either demand excessive training resources or introduce significant inference latency. In this paper, we propose GFlOwNet-guided distribution AlignmenT (GOAT) for LM-based TTS, a post-training framework that mitigates hallucinations without relying on massive resources or inference cost. Specifically, we first conduct an uncertainty analysis, revealing a strong positive correlation between hallucination and model uncertainty. Based on this, we reformulate TTS generation as a trajectory flow optimization problem and introduce an enhanced Subtrajectory Balance objective together with a sharpened internal reward as target distribution. We further integrate reward temperature decay and learning rate optimization for stability and performance balance. Extensive experiments show that GOAT reduce over 50% character error rates on challenging test cases and lowering uncertainty by up to 58%, demonstrating its strong generalization ability and effectiveness.