Developing a Multilingual Dataset and Evaluation Metrics for Code-Switching: A Focus on Hong Kong's Polylingual Dynamics
This work addresses the problem of evaluating ASR systems in code-switching scenarios for multilingual regions like Hong Kong, though it is incremental as it builds on existing models and metrics.
The authors tackled the lack of datasets for code-switching in multilingual communities by developing a 34.8-hour Mixed Cantonese and English audio dataset and fine-tuning Whisper ASR, achieving impressive zero-shot performance, and introduced a novel FAL evaluation metric to address limitations in traditional ASR metrics.
The existing audio datasets are predominantly tailored towards single languages, overlooking the complex linguistic behaviors of multilingual communities that engage in code-switching. This practice, where individuals frequently mix two or more languages in their daily interactions, is particularly prevalent in multilingual regions such as Hong Kong, China. To bridge this gap, we have developed a 34.8-hour dataset of Mixed Cantonese and English (MCE) audio using our Multi-Agent Data Generation Framework (MADGF). We fine-tuned the open-source multilingual Automatic Speech Recognition (ASR) model, Whisper, with the MCE dataset, leading to impressive zero-shot performance. The traditional metrics overlook important factors such as latency in real-world applications and code-switching scenarios. We have introduced a novel evaluation metric called Fidelity to the Original Audio, Accuracy, and Latency (FAL). This metric aims to overcome the limitations of traditional metrics used to assess ASR systems.