asr_eval: Algorithms and tools for multi-reference and streaming speech recognition evaluation
This work addresses evaluation challenges in speech recognition, particularly for non-Latin languages and streaming scenarios, but it is incremental as it builds on existing evaluation methods.
The authors tackled the problem of evaluating speech recognition systems by introducing a new string alignment algorithm for multi-reference labeling and better word alignment, and they demonstrated that models can adapt to dataset-specific labeling, creating an illusion of metric improvement. They also collected a novel test set for Russian speech and provided tools for streaming evaluation.
We propose several improvements to the speech recognition evaluation. First, we propose a string alignment algorithm that supports both multi-reference labeling, arbitrary-length insertions and better word alignment. This is especially useful for non-Latin languages, those with rich word formation, to label cluttered or longform speech. Secondly, we collect a novel test set DiverseSpeech-Ru of longform in-the-wild Russian speech with careful multi-reference labeling. We also perform multi-reference relabeling of popular Russian tests set and study fine-tuning dynamics on its corresponding train set. We demonstrate that the model often adopts to dataset-specific labeling, causing an illusion of metric improvement. Based on the improved word alignment, we develop tools to evaluate streaming speech recognition and to align multiple transcriptions to compare them visually. Additionally, we provide uniform wrappers for many offline and streaming speech recognition models. Our code will be made publicly available.