Kakugo: Distillation of Low-Resource Languages into Small Language Models
This provides an accessible method for communities to develop AI for low-resource languages, though it is incremental as it builds on existing distillation techniques.
The authors tackled the problem of training small language models for low-resource languages by developing Kakugo, a pipeline that uses a teacher model to generate synthetic data, resulting in improved performance across NLP tasks for 54 languages at a cost under $50 per language.
We present Kakugo, a novel and cost-effective pipeline designed to train general-purpose Small Language Models (SLMs) for low-resource languages using only the language name as input. By using a large teacher model to generate synthetic prompts and translate instruction datasets, we produced training data and SLMs for 54 low-resource languages. Evaluations across a diverse set of general natural language processing tasks, including translation, classification, and question answering, demonstrate that our pipeline consistently improves performance over base models. With a total generation and training cost of under $50 per language, Kakugo offers an accessible method for communities to develop language-specific AI.