ConLID: Supervised Contrastive Learning for Low-Resource Language Identification
This addresses language identification challenges for low-resource languages in multilingual LLM pretraining, though it appears incremental as it builds on existing contrastive learning methods.
The paper tackles the problem of poor language identification performance for low-resource languages, which often rely on single-domain data like the Bible, by proposing a supervised contrastive learning approach that improves out-of-domain performance by 3.2%.
Language identification (LID) is a critical step in curating multilingual LLM pretraining corpora from web crawls. While many studies on LID model training focus on collecting diverse training data to improve performance, low-resource languages -- often limited to single-domain data, such as the Bible -- continue to perform poorly. To resolve these class imbalance and bias issues, we propose a novel supervised contrastive learning (SCL) approach to learn domain-invariant representations for low-resource languages. Through an extensive analysis, we show that our approach improves LID performance on out-of-domain data for low-resource languages by 3.2%, demonstrating its effectiveness in enhancing LID models.