Combining Contrastive and Non-Contrastive Losses for Fine-Tuning Pretrained Models in Speech Analysis
This work addresses the problem of ineffective fine-tuning for paralinguistic speech analysis, which is incremental as it builds on existing pretraining methods with a novel loss combination.
The paper tackles the challenge of fine-tuning pretrained speech models for paralinguistic tasks with limited data and high class variance by proposing a two-step approach that combines contrastive and non-contrastive losses to improve embedding invariance and discriminability, resulting in consistent outperformance of baselines and surpassing state-of-the-art on emotion classification.
Embedding paralinguistic properties is a challenging task as there are only a few hours of training data available for domains such as emotional speech. One solution to this problem is to pretrain a general self-supervised speech representation model on large amounts of unlabeled speech. This pretrained model is then finetuned to a specific task. Paralinguistic properties however have notoriously high class variance, making the finetuning ineffective. In this work, we propose a two step approach to this. First we improve the embedding space, then we train an adapter to bridge the gap from the embedding space to a classification task. In order to improve the class invariance we use a combination of contrastive and non-contrastive losses to explicitly optimize for class invariant, yet discriminative features. Our approach consistently outperforms baselines that are finetuned end-to-end on multiple tasks and surpasses a benchmark on state-of-the-art emotion classification.