Position-invariant Fine-tuning of Speech Enhancement Models with Self-supervised Speech Representations
This addresses a general limitation in fine-tuning self-supervised speech models for noisy conditions, with incremental improvements for speech enhancement applications.
The paper tackled the problem of speech enhancement models exploiting positional embeddings in self-supervised learning representations during fine-tuning, which reduces content-related learning. The result showed that using speed perturbations with a soft-DTW loss achieved faster convergence and improved downstream performance compared to standard methods.
Integrating front-end speech enhancement (SE) models with self-supervised learning (SSL)-based speech models is effective for downstream tasks in noisy conditions. SE models are commonly fine-tuned using SSL representations with mean squared error (MSE) loss between enhanced and clean speech. However, MSE is prone to exploiting positional embeddings in SSL models, allowing the objective to be minimised through positional correlations instead of content-related information. This work frames the problem as a general limitation of self-supervised representation fine-tuning and investigates it through representation-guided SE. Two strategies are considered: (1) zero-padding, previously explored in SSL pre-training but here examined in the fine-tuning setting, and (2) speed perturbations with a soft-DTW loss. Experiments show that the soft-DTW-based approach achieves faster convergence and improved downstream performance, underscoring the importance of position-invariant fine-tuning in SSL-based speech modelling.