Modeling Dependent Structure for Utterances in ASR Evaluation
This work addresses a methodological bottleneck in ASR evaluation for researchers, though it is incremental as it builds on existing blockwise bootstrap approaches.
The paper tackles the challenge of modeling dependent structure among utterances for accurate significance analysis of word error rate (WER) in ASR evaluation, presenting graphical lasso-based methods to estimate uncorrelated blocks and showing statistical consistency and validity on the LibriSpeech dataset.
The bootstrap resampling method has been popular for performing significance analysis on word error rate (WER) in automatic speech recognition (ASR) evaluation. To deal with dependent speech data, the blockwise bootstrap approach is also introduced. By dividing utterances into uncorrelated blocks, this approach resamples these blocks instead of original data. However, it is typically nontrivial to uncover the dependent structure among utterances and identify the blocks, which might lead to subjective conclusions in statistical testing. In this paper, we present graphical lasso based methods to explicitly model such dependency and estimate uncorrelated blocks of utterances in a rigorous way, after which blockwise bootstrap is applied on top of the inferred blocks. We show the resulting variance estimator of WER in ASR evaluation is statistically consistent under mild conditions. We also demonstrate the validity of proposed approach on LibriSpeech dataset.