Model-Based Approach for Measuring the Fairness in ASR
This work addresses fairness measurement for ASR systems, which is crucial for ensuring equitable performance across diverse populations, though it is incremental as it applies an existing statistical method to this domain.
The paper tackled the problem of measuring fairness in automatic speech recognition (ASR) systems by addressing issues like controlling nuisance factors and handling unobserved heterogeneity across speakers, introducing a mixed-effects Poisson regression method that effectively measures and interprets word error rate (WER) differences among subgroups, as demonstrated on synthetic and real-world data.
The issue of fairness arises when the automatic speech recognition (ASR) systems do not perform equally well for all subgroups of the population. In any fairness measurement studies for ASR, the open questions of how to control the nuisance factors, how to handle unobserved heterogeneity across speakers, and how to trace the source of any word error rate (WER) gap among different subgroups are especially important - if not appropriately accounted for, incorrect conclusions will be drawn. In this paper, we introduce mixed-effects Poisson regression to better measure and interpret any WER difference among subgroups of interest. Particularly, the presented method can effectively address the three problems raised above and is very flexible to use in practical disparity analyses. We demonstrate the validity of proposed model-based approach on both synthetic and real-world speech data.