Speech Synthesis By Unrolling Diffusion Process using Neural Network Layers
This work addresses the problem of slow and distorted speech synthesis for real-time applications, offering an incremental improvement over existing diffusion models.
The paper tackles the slow reverse diffusion process in speech synthesis by introducing UDPNet, which unrolls the diffusion steps into network layers and predicts latent variables, resulting in improved quality and efficiency over state-of-the-art methods on single- and multi-speaker datasets.
This work introduces UDPNet, a novel architecture designed to accelerate the reverse diffusion process in speech synthesis. Unlike traditional diffusion models that rely on timestep embeddings and shared network parameters, UDPNet unrolls the reverse diffusion process directly into the network architecture, with successive layers corresponding to equally spaced steps in the diffusion schedule. Each layer progressively refines the noisy input, culminating in a high-fidelity estimation of the original data, \(x_0\). Additionally, we redefine the learning target by predicting latent variables instead of the conventional \(x_0\) or noise \(ε_0\). This shift addresses the common issue of large prediction errors in early denoising stages, effectively reducing speech distortion. Extensive evaluations on single- and multi-speaker datasets demonstrate that UDPNet consistently outperforms state-of-the-art methods in both quality and efficiency, while generalizing effectively to unseen speech. These results position UDPNet as a robust solution for real-time speech synthesis applications. Sample audio is available at https://onexpeters.github.io/UDPNet.