Improving Fairness and Robustness in End-to-End Speech Recognition through unsupervised clustering
This addresses fairness and robustness problems in ASR systems for diverse user groups, representing an incremental improvement over existing methods.
The paper tackles fairness and robustness issues in end-to-end speech recognition systems by developing a privacy-preserving approach that uses unsupervised clustering of acoustic features instead of direct speaker embeddings, resulting in improved performance across all demographic groups, particularly for different accents.
The challenge of fairness arises when Automatic Speech Recognition (ASR) systems do not perform equally well for all sub-groups of the population. In the past few years there have been many improvements in overall speech recognition quality, but without any particular focus on advancing Equality and Equity for all user groups for whom systems do not perform well. ASR fairness is therefore also a robustness issue. Meanwhile, data privacy also takes priority in production systems. In this paper, we present a privacy preserving approach to improve fairness and robustness of end-to-end ASR without using metadata, zip codes, or even speaker or utterance embeddings directly in training. We extract utterance level embeddings using a speaker ID model trained on a public dataset, which we then use in an unsupervised fashion to create acoustic clusters. We use cluster IDs instead of speaker utterance embeddings as extra features during model training, which shows improvements for all demographic groups and in particular for different accents.