CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing
This addresses the need for more efficient and adaptable speech processing models, particularly for under-resourced and unseen tasks, with incremental improvements over standard fine-tuning methods.
The paper tackled the problem of improving speech processing tasks by introducing CA-SSLR, a condition-aware self-supervised learning representation that integrates language and speaker embeddings to reduce reliance on audio features, achieving a 10% reduction in LID errors, 37% improvement in ASR CER, and 27% decrease in SV EER.
We introduce Condition-Aware Self-Supervised Learning Representation (CA-SSLR), a generalist conditioning model broadly applicable to various speech-processing tasks. Compared to standard fine-tuning methods that optimize for downstream models, CA-SSLR integrates language and speaker embeddings from earlier layers, making the SSL model aware of the current language and speaker context. This approach reduces the reliance on input audio features while preserving the integrity of the base SSLR. CA-SSLR improves the model's capabilities and demonstrates its generality on unseen tasks with minimal task-specific tuning. Our method employs linear modulation to dynamically adjust internal representations, enabling fine-grained adaptability without significantly altering the original model behavior. Experiments show that CA-SSLR reduces the number of trainable parameters, mitigates overfitting, and excels in under-resourced and unseen tasks. Specifically, CA-SSLR achieves a 10% relative reduction in LID errors, a 37% improvement in ASR CER on the ML-SUPERB benchmark, and a 27% decrease in SV EER on VoxCeleb-1, demonstrating its effectiveness.