Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metric
This addresses the issue of poor correlation between WER and user perception for ASR systems, offering a more accurate evaluation method for voice-driven applications.
They tackled the problem of evaluating ASR system quality by proposing SemDist, a semantic distance metric, which achieved higher correlation with user perception and downstream NLU tasks compared to traditional WER.
Measuring automatic speech recognition (ASR) system quality is critical for creating user-satisfying voice-driven applications. Word Error Rate (WER) has been traditionally used to evaluate ASR system quality; however, it sometimes correlates poorly with user perception/judgement of transcription quality. This is because WER weighs every word equally and does not consider semantic correctness which has a higher impact on user perception. In this work, we propose evaluating ASR output hypotheses quality with SemDist that can measure semantic correctness by using the distance between the semantic vectors of the reference and hypothesis extracted from a pre-trained language model. Our experimental results of 71K and 36K user annotated ASR output quality show that SemDist achieves higher correlation with user perception than WER. We also show that SemDist has higher correlation with downstream Natural Language Understanding (NLU) tasks than WER.