A Study of Gender Impact in Self-supervised Models for Speech-to-Text Systems
This addresses fairness and performance issues in speech-to-text systems for users and developers, but it is incremental as it builds on existing models and focuses on a specific language.
The study investigated how gender distribution in pre-training data affects self-supervised models like wav2vec 2.0 for speech-to-text tasks in French, finding that gender-specific pre-training lowers performance in end-to-end ASR but leads to complex patterns in feature extraction, with no strong variation in fairness metrics.
Self-supervised models for speech processing emerged recently as popular foundation blocks in speech processing pipelines. These models are pre-trained on unlabeled audio data and then used in speech processing downstream tasks such as automatic speech recognition (ASR) or speech translation (ST). Since these models are now used in research and industrial systems alike, it becomes necessary to understand the impact caused by some features such as gender distribution within pre-training data. Using French as our investigation language, we train and compare gender-specific wav2vec 2.0 models against models containing different degrees of gender balance in their pre-training data. The comparison is performed by applying these models to two speech-to-text downstream tasks: ASR and ST. Results show the type of downstream integration matters. We observe lower overall performance using gender-specific pre-training before fine-tuning an end-to-end ASR system. However, when self-supervised models are used as feature extractors, the overall ASR and ST results follow more complex patterns in which the balanced pre-trained model does not necessarily lead to the best results. Lastly, our crude 'fairness' metric, the relative performance difference measured between female and male test sets, does not display a strong variation from balanced to gender-specific pre-trained wav2vec 2.0 models.