Joint Residual Reweighting for Classifier Free Guidance in Flow-Matching Zero-Shot TTS
For researchers in zero-shot TTS, this work offers a method to better balance speaker fidelity and text accuracy, though it is an incremental improvement over existing branch-selective guidance methods.
The paper addresses the trade-off between text correctness and speaker similarity in flow-matching zero-shot TTS. By decomposing the CFG field into text, speaker, and joint residuals and proposing joint residual reweighting, they improve speaker similarity while maintaining text correctness, as shown on F5-TTS and CosyVoice2.
Classifier-free guidance (CFG) is widely used in flow-matching-based zero-shot text-to-speech (TTS), where generation is typically controlled by two conditions: the target text and a prompt speech signal. Standard CFG strengthens these conditions jointly, while recent branch-selective guidance methods attempt to enhance text or speaker conditioning separately, often leading to a trade-off between text correctness and speaker similarity. In this paper, we revisit the CFG under independently masked text and speech-prompt conditions, and decompose the guidance field into text, speaker, and joint residuals. We show that conventional speaker-selective guidance entangles the speaker residual with the joint residual, which may disturb text-related generation. Based on this observation, we propose joint residual reweighting, which independently controls the speaker and joint residuals within the standard CFG framework. Experiments on F5-TTS and CosyVoice2 show that the proposed method improves speaker similarity while maintaining competitive text correctness, demonstrating the usefulness of the joint residual for balancing speaker fidelity and text accuracy in zero-shot TTS.