Evaluating ASR Confidence Scores for Automated Error Detection in User-Assisted Correction Interfaces
This work addresses the problem of improving user interaction and explainability in ASR systems for users needing manual correction, but it is incremental as it highlights limitations of existing confidence scores.
The study tackled the problem of using ASR confidence scores for automated error detection in user-assisted correction interfaces, finding that while scores correlate with accuracy, their error detection performance is limited, with classifiers missing errors or generating false positives, and they did not improve correction efficiency or user perception.
Despite advances in Automatic Speech Recognition (ASR), transcription errors persist and require manual correction. Confidence scores, which indicate the certainty of ASR results, could assist users in identifying and correcting errors. This study evaluates the reliability of confidence scores for error detection through a comprehensive analysis of end-to-end ASR models and a user study with 36 participants. The results show that while confidence scores correlate with transcription accuracy, their error detection performance is limited. Classifiers frequently miss errors or generate many false positives, undermining their practical utility. Confidence-based error detection neither improved correction efficiency nor was perceived as helpful by participants. These findings highlight the limitations of confidence scores and the need for more sophisticated approaches to improve user interaction and explainability of ASR results.