DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switching
This addresses the need for more robust LID systems in real-world multilingual NLP applications, though it is incremental as it focuses on evaluation and benchmarking.
The paper tackled the problem of language identification (LID) systems overfitting to clean data by evaluating state-of-the-art models across diverse domains and code-switched text, finding that performance degrades sharply on noisy inputs and existing models struggle with multiple languages in sentences.
Language Identification (LID) is a core task in multilingual NLP, yet current systems often overfit to clean, monolingual data. This work introduces DIVERS-BENCH, a comprehensive evaluation of state-of-the-art LID models across diverse domains, including speech transcripts, web text, social media texts, children's stories, and code-switched text. Our findings reveal that while models achieve high accuracy on curated datasets, performance degrades sharply on noisy and informal inputs. We also introduce DIVERS-CS, a diverse code-switching benchmark dataset spanning 10 language pairs, and show that existing models struggle to detect multiple languages within the same sentence. These results highlight the need for more robust and inclusive LID systems in real-world settings.