RAS: a Reliability Oriented Metric for Automatic Speech Recognition
For users and downstream applications of ASR, this work addresses the problem of overconfident incorrect transcriptions by enabling models to abstain from uncertain segments.
The paper introduces RAS, a reliability-oriented metric for ASR that balances transcription informativeness and error aversion, and trains an abstention-aware ASR model that improves transcription reliability while maintaining competitive accuracy.
Automatic speech recognition systems often produce confident yet incorrect transcriptions under noisy or ambiguous conditions, which can be misleading for both users and downstream applications. Standard evaluation based on Word Error Rate focuses solely on accuracy and fails to capture transcription reliability. We introduce an abstention-aware transcription framework that enables ASR models to explicitly abstain from uncertain segments. To evaluate reliability under abstention, we propose RAS, a reliability-oriented metric that balances transcription informativeness and error aversion, with its trade-off parameter calibrated by human preference. We then train an abstention-aware ASR model through supervised bootstrapping followed by reinforcement learning. Our experiments demonstrate substantial improvements in transcription reliability while maintaining competitive accuracy.