UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages
This work addresses the under-performance of LLMs on low-resource languages, providing an affordable approach for adaptation using consumer hardware, though it is incremental as it builds on existing data sources and adapter methods.
The paper tackles the problem of limited training data for low-resource languages in large language models by presenting UnifiedCrawl, a method to efficiently collect large mono-lingual datasets from Common Crawl, which significantly boosts performance on low-resource languages through fine-tuning with minimal compute resources.
Large language models (LLMs) under-perform on low-resource languages due to limited training data. We present a method to efficiently collect text data for low-resource languages from the entire Common Crawl corpus. Our approach, UnifiedCrawl, filters and extracts common crawl using minimal compute resources, yielding mono-lingual datasets much larger than previously available sources. We demonstrate that leveraging this data to fine-tuning multilingual LLMs via efficient adapter methods (QLoRA) significantly boosts performance on the low-resource language, while minimizing VRAM usage. Our experiments show large improvements in language modeling perplexity and an increase in few-shot prompting scores. Our work and released source code provide an affordable approach to improve LLMs for low-resource languages using consumer hardware. Our source code is available here at https://github.com/bethelmelesse/unifiedcrawl.