Enhancing Flow Matching with A Unified Guidance Framework for Efficient and Robust Speech Synthesis
For speech synthesis researchers and practitioners, this work addresses efficiency and robustness bottlenecks in Flow Matching, offering a practical acceleration without quality loss.
Flow Matching for speech synthesis suffers from high inference latency and timbre leakage. The proposed unified guidance framework with data and model guidance achieves nearly 3x faster inference and improved speaker similarity over SOTA baselines.
Flow Matching (FM) has emerged as a powerful paradigm for speech generation but remains constrained by high inference latency and timbre leakage. To address these bottlenecks, we propose a unified guidance framework that enhances generation efficiency and robustness through two complementary strategies. On the data front, we introduce Data-guidance via heterogeneous augmentation, encouraging the model to disentangle linguistic content from acoustic residue. In parallel, we propose an enhanced Model-guidance mechanism that synergizes trajectory rectification with a novel intrinsic guidance objective. This approach distills conditional knowledge into network weights and straightens inference trajectory path, thereby eliminating Classifier-Free Guidance (CFG) overhead. Experiments demonstrate that our framework accelerates inference by nearly three times while effectively improving speaker similarity compared to state-of-the-art baselines.