PIER: A Novel Metric for Evaluating What Matters in Code-Switching
This addresses the challenge of accurately assessing model performance in code-switching for speech recognition, though it is incremental as it modifies an existing metric rather than introducing a new method.
The paper tackles the problem of evaluating Automatic Speech Recognition on code-switching by showing that standard metrics like Word-Error-Rate improve with fine-tuning on non-code-switched data, but code-switched words worsen, leading to the proposal of PIER, a variant focusing on specific words of interest, which reveals significant room for improvement.
Code-switching, the alternation of languages within a single discourse, presents a significant challenge for Automatic Speech Recognition. Despite the unique nature of the task, performance is commonly measured with established metrics such as Word-Error-Rate (WER). However, in this paper, we question whether these general metrics accurately assess performance on code-switching. Specifically, using both Connectionist-Temporal-Classification and Encoder-Decoder models, we show fine-tuning on non-code-switched data from both matrix and embedded language improves classical metrics on code-switching test sets, although actual code-switched words worsen (as expected). Therefore, we propose Point-of-Interest Error Rate (PIER), a variant of WER that focuses only on specific words of interest. We instantiate PIER on code-switched utterances and show that this more accurately describes the code-switching performance, showing huge room for improvement in future work. This focused evaluation allows for a more precise assessment of model performance, particularly in challenging aspects such as inter-word and intra-word code-switching.