Is Word Error Rate a good evaluation metric for Speech Recognition in Indic Languages?
This addresses the issue of inaccurate evaluation for ASR in languages with complex pronunciation and multiple writing forms, though it is incremental as it builds on existing methods like wav2vec 2.0.
The paper tackles the problem of evaluating speech recognition in Indic languages by proposing new metrics, Alternate Word Error Rate (AWER) and Alternate Character Error Rate (ACER), which account for half-characters and multiple writing forms, showing improvements of up to 3% in AWER and 7% in ACER for Hindi.
We propose a new method for the calculation of error rates in Automatic Speech Recognition (ASR). This new metric is for languages that contain half characters and where the same character can be written in different forms. We implement our methodology in Hindi which is one of the main languages from Indic context and we think this approach is scalable to other similar languages containing a large character set. We call our metrics Alternate Word Error Rate (AWER) and Alternate Character Error Rate (ACER). We train our ASR models using wav2vec 2.0\cite{baevski2020wav2vec} for Indic languages. Additionally we use language models to improve our model performance. Our results show a significant improvement in analyzing the error rates at word and character level and the interpretability of the ASR system is improved upto $3$\% in AWER and $7$\% in ACER for Hindi. Our experiments suggest that in languages which have complex pronunciation, there are multiple ways of writing words without changing their meaning. In such cases AWER and ACER will be more useful rather than WER and CER as metrics. Further, we open source a new benchmarking dataset of 21 hours for Hindi with the new metric scripts.