SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluation
This addresses the need for impartial and replicable evaluation of ASR systems, which is crucial for researchers and developers in speech technology, though it is incremental as it builds on existing evaluation methods.
The paper introduces the SpeechColab Leaderboard, an open-source platform for evaluating Automatic Speech Recognition (ASR) systems, providing a comprehensive benchmark of state-of-the-art models and quantifying how scoring nuances affect results. It also proposes a modified Token-Error-Rate (mTER) metric, showing robustness and backward compatibility compared to TER.
In the wake of the surging tide of deep learning over the past decade, Automatic Speech Recognition (ASR) has garnered substantial attention, leading to the emergence of numerous publicly accessible ASR systems that are actively being integrated into our daily lives. Nonetheless, the impartial and replicable evaluation of these ASR systems encounters challenges due to various crucial subtleties. In this paper we introduce the SpeechColab Leaderboard, a general-purpose, open-source platform designed for ASR evaluation. With this platform: (i) We report a comprehensive benchmark, unveiling the current state-of-the-art panorama for ASR systems, covering both open-source models and industrial commercial services. (ii) We quantize how distinct nuances in the scoring pipeline influence the final benchmark outcomes. These include nuances related to capitalization, punctuation, interjection, contraction, synonym usage, compound words, etc. These issues have gained prominence in the context of the transition towards an End-to-End future. (iii) We propose a practical modification to the conventional Token-Error-Rate (TER) evaluation metric, with inspirations from Kolmogorov complexity and Normalized Information Distance (NID). This adaptation, called modified-TER (mTER), achieves proper normalization and symmetrical treatment of reference and hypothesis. By leveraging this platform as a large-scale testing ground, this study demonstrates the robustness and backward compatibility of mTER when compared to TER. The SpeechColab Leaderboard is accessible at https://github.com/SpeechColab/Leaderboard