Repurposing a Speech Classifier for Guided Diffusion-Based Speech Generation
This work offers a more compact alternative to classifier-guided diffusion for speech generation, reducing the need for separate classifier and diffusion models.
The authors repurpose a pretrained speech classifier as the backbone for diffusion-based speech generation, attaching a lightweight subnetwork trained under Denoising Score Matching. This achieves high speech quality with reduced memory and computation compared to standard classifier guidance.
Classifier guidance is a way to control diffusion generation by using a noise-conditioned classifier to steer the sampling process toward a target class. One drawback of classifier guidance is that it requires two separately trained models: a classifier and a diffusion model. We therefore study a more compact alternative in which a conventionally trained speech classifier is repurposed as the backbone for diffusion generation. Starting from a frozen noise-conditioned classifier in log-Mel space, we attach a lightweight subnetwork that reuses intermediate classifier representations and train only this subnetwork under a Denoising Score Matching objective. Our work shows that a pretrained classifier can be repurposed for conditional generation, providing an appealing bridge between discriminative modeling and conditional speech synthesis resulting in high speech quality within a single-backbone model, with reduced memory footprint and computational cost.