H_eval: A new hybrid evaluation metric for automatic speech recognition tasks
This work addresses the problem of evaluating ASR systems more effectively for researchers and practitioners, though it appears incremental as it builds on prior semantic metrics.
The paper tackles the limitations of existing evaluation metrics for automatic speech recognition (ASR) by proposing H_eval, a hybrid metric that combines semantic correctness and error rate, achieving a 49 times reduction in computation time compared to BERTScore and strong correlation with downstream NLP tasks.
Many studies have examined the shortcomings of word error rate (WER) as an evaluation metric for automatic speech recognition (ASR) systems. Since WER considers only literal word-level correctness, new evaluation metrics based on semantic similarity such as semantic distance (SD) and BERTScore have been developed. However, we found that these metrics have their own limitations, such as a tendency to overly prioritise keywords. We propose H_eval, a new hybrid evaluation metric for ASR systems that considers both semantic correctness and error rate and performs significantly well in scenarios where WER and SD perform poorly. Due to lighter computation compared to BERTScore, it offers 49 times reduction in metric computation time. Furthermore, we show that H_eval correlates strongly with downstream NLP tasks. Also, to reduce the metric calculation time, we built multiple fast and lightweight models using distillation techniques