VARAN: Variational Inference for Self-Supervised Speech Models Fine-Tuning on Downstream Tasks
This work addresses a bottleneck in adapting self-supervised speech models for downstream tasks, offering an incremental improvement in efficiency and flexibility for speech processing applications.
The paper tackles the problem of static layer aggregation in fine-tuned self-supervised speech models by proposing VARAN, a framework that dynamically tailors layer aggregation to individual inputs, resulting in superior performance on tasks like automatic speech recognition and speech emotion recognition, especially with LoRA fine-tuning.
Conventional methods for aggregating layers in fine-tuned self-supervised speech models, such as using the final layer or weighted sum, suffer from information bottlenecks and static feature weighting for all dataset examples. We propose VARAN, a framework that dynamically tailors layer aggregation to individual inputs. By employing layer-specialized probing heads and data-dependent weighting, VARAN adaptively prioritizes layer's features based on input. Evaluations on automatic speech recognition and speech emotion recognition tasks demonstrate VARAN's superior performance, particularly when using the LoRA fine-tuning technique. The framework resolves the trade-off between preserving layer-specific information and enabling flexible feature utilization, advancing efficient adaptation of self-supervised speech representations.