BERnaT: Basque Encoders for Representing Natural Textual Diversity
This work addresses representational biases in language models for low-resource languages like Basque, though it is incremental as it applies existing methods to new data.
The paper tackled the problem of language models excluding non-standard linguistic varieties by constructing diverse corpora for Basque and pre-training encoder models, resulting in models that consistently outperformed standard-only models across all task types without compromising benchmark accuracy.
Language models depend on massive text corpora that are often filtered for quality, a process that can unintentionally exclude non-standard linguistic varieties, reduce model robustness and reinforce representational biases. In this paper, we argue that language models should aim to capture the full spectrum of language variation (dialectal, historical, informal, etc.) rather than relying solely on standardized text. Focusing on Basque, a morphologically rich and low-resource language, we construct new corpora combining standard, social media, and historical sources, and pre-train the BERnaT family of encoder-only models in three configurations: standard, diverse, and combined. We further propose an evaluation framework that separates Natural Language Understanding (NLU) tasks into standard and diverse subsets to assess linguistic generalization. Results show that models trained on both standard and diverse data consistently outperform those trained on standard corpora, improving performance across all task types without compromising standard benchmark accuracy. These findings highlight the importance of linguistic diversity in building inclusive, generalizable language models.