Curió-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretraining
This work addresses the challenge of efficient language model adaptation for specific domains, showing that data quality can outweigh quantity, which is incremental but impactful for practitioners in NLP and education.
The study tackled the problem of adapting language models to new linguistic contexts through continued pretraining, finding that a variant trained on a smaller, high-quality educational subset (10 billion tokens) outperformed the full-corpus model (100 billion tokens) despite using only 10% of the data and 20% of the computation.
Continued pretraining extends a language model's capabilities by further exposing it to additional data, often tailored to a specific linguistic or domain context. This strategy has emerged as an efficient alternative to full retraining when adapting general-purpose models to new settings. In this work, we investigate this paradigm through Curió 7B, a 7-billion-parameter model derived from LLaMA-2 and trained on 100 billion Portuguese tokens from the ClassiCC-PT corpus - the most extensive Portuguese-specific continued-pretraining effort above the three-billion-parameter scale to date. Beyond scale, we investigate whether quantity alone suffices or whether data quality plays a decisive role in linguistic adaptation. To this end, we introduce Curió-Edu 7B, a variant trained exclusively on the educational and STEM-filtered subset of the same corpus, totaling just 10 billion tokens. Despite using only 10% of the data and 20% of the computation, Curió-Edu 7B surpasses the full-corpus model in our evaluations, demonstrating that data selection can be fundamental even when adapting models with limited prior exposure to the target language. The developed models are available at https://huggingface.co/collections/ClassiCC-Corpus/curio-edu