Towards Robust Generative Speech Enhancement Using Vector Quantisation-Based Neural Audio Codec
For researchers in speech enhancement, this work provides insights into the role of VQ regularisation and demonstrates that continuous latent modeling can outperform discrete token-based methods, but the findings are incremental as they build on existing NAC frameworks.
This work investigates continuous and discrete latent space modeling in vector quantisation (VQ)-based neural audio codec (NAC) speech enhancement, showing that a fully fine-tuned continuous model (cNAC-SE) outperforms all discrete variants and achieves leading DNS-MOS scores among generative approaches. VQ regularisation is found to enhance robustness via clean-prior-constrained regularisation, independent of discrete token processing.
This work investigates modelling strategies in continuous and discrete latent spaces in the vector quantisation (VQ)-based neural audio codec (NAC) speech enhancement (SE), along with the role of VQ regularisation. We propose cNAC-SE and dNAC-SE frameworks that predict continuous representations and discrete tokens in latent space, respectively. Theoretical analysis and visualisations in latent space are performed to exhibit their inherent modelling mechanisms. Experimental results show that the fully fine-tuned cNAC-SE model consistently outperforms all dNAC-SE variants across diverse test conditions and achieves leading performance among established generative approaches in DNS-MOS metrics. Comparison with the discriminative counterpart shows that VQ enhances robustness through an intrinsic effect of clean-prior-constrained regularisation, independent of discrete token processing. This highlights the transferable value of VQ regularisation to other continuous modelling methods.