CHALIS: A Challenge Dataset for Language Identification in Difficult Scenarios
For researchers and practitioners in language identification, this dataset exposes critical failure modes in current systems, enabling targeted improvements.
The paper introduces CHALIS, a benchmark dataset for language identification in difficult scenarios involving cousin languages and orthographic noise, and shows that existing systems struggle significantly, especially with lower-resource languages and transliterated text.
We present CHALIS (Challenging Language Identification Samples), a new benchmark dataset explicitly designed to address difficult cases in language identification: cousin languages and orthographic noise. Our dataset has two parts: First, we collected sentences shared across mutually intelligible language pairs (Czech/Slovak, Spanish/Catalan, Portuguese/Galician, Danish/Norwegian). The second part tests for orthography noise: we transliterate text across multiple scripts, remove diacritics, simulate homoglyph attacks, and use Internet slang. We evaluate four widely used language identification systems on CHALIS and demonstrate that all struggle substantially in these scenarios, especially on lower-resource languages within cousin pairs and on transliterated input. The resource is publicly available at https://huggingface.co/datasets/michal-tichy/CHALIS.