Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus
This addresses the challenge of building large multilingual text corpora for NLP, which is crucial for advancing language technologies, but the work is incremental as it improves on existing LangID methods rather than introducing a new paradigm.
The paper tackled the problem of low accuracy in automatic language identification (LangID) for web-crawl text corpora, especially for lower-resource languages, where human-judged accuracy was only around 5%, and proposed techniques that increased median dataset precision from 5.5% to 71.2%.
Large text corpora are increasingly important for a wide variety of Natural Language Processing (NLP) tasks, and automatic language identification (LangID) is a core technology needed to collect such datasets in a multilingual context. LangID is largely treated as solved in the literature, with models reported that achieve over 90% average F1 on as many as 1,366 languages. We train LangID models on up to 1,629 languages with comparable quality on held-out test sets, but find that human-judged LangID accuracy for web-crawl text corpora created using these models is only around 5% for many lower-resource languages, suggesting a need for more robust evaluation. Further analysis revealed a variety of error modes, arising from domain mismatch, class imbalance, language similarity, and insufficiently expressive models. We propose two classes of techniques to mitigate these errors: wordlist-based tunable-precision filters (for which we release curated lists in about 500 languages) and transformer-based semi-supervised LangID models, which increase median dataset precision from 5.5% to 71.2%. These techniques enable us to create an initial data set covering 100K or more relatively clean sentences in each of 500+ languages, paving the way towards a 1,000-language web text corpus.