Style-agnostic evaluation of ASR using multiple reference transcripts
This addresses evaluation biases in ASR for researchers and practitioners, though it is incremental as it builds on existing multi-reference approaches.
The paper tackled the limitations of Word Error Rate (WER) in speech recognition evaluation by using multiple reference transcripts to perform style-agnostic assessment, finding that existing WER reports likely over-estimate contentful errors by state-of-the-art ASR systems.
Word error rate (WER) as a metric has a variety of limitations that have plagued the field of speech recognition. Evaluation datasets suffer from varying style, formality, and inherent ambiguity of the transcription task. In this work, we attempt to mitigate some of these differences by performing style-agnostic evaluation of ASR systems using multiple references transcribed under opposing style parameters. As a result, we find that existing WER reports are likely significantly over-estimating the number of contentful errors made by state-of-the-art ASR systems. In addition, we have found our multireference method to be a useful mechanism for comparing the quality of ASR models that differ in the stylistic makeup of their training data and target task.