Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation
This addresses the problem of limited multilingual instruction tuning for researchers and practitioners, though it is incremental as it builds on existing LoRA and instruction-tuning methods.
The authors tackled the scarcity of high-quality multilingual instruction-response datasets by creating Bactrian-X, a dataset of 3.4 million pairs across 52 languages, and used it with LoRA adapters to train models that outperform vanilla and existing instruction-tuned models in multilingual evaluations.
Instruction tuning has shown great promise in improving the performance of large language models. However, research on multilingual instruction tuning has been limited due to the scarcity of high-quality instruction-response datasets across different languages. To bridge this gap, we present Bactrian-X, a comprehensive multilingual parallel dataset of 3.4 million instruction-response pairs across 52 languages. Leveraging this dataset, we train a set of adapters using low-rank adaptation (LoRA), which are lightweight components that seamlessly integrate with large language models. These adapters have a substantially lower parameter count than the base model, making them easily replaceable and usable as plug-ins for different languages or language groups. Extensive experiments in various multilingual evaluation settings demonstrate that models derived from LoRA-based training over Bactrian-X outperform both the vanilla models and existing instruction-tuned models. The code and models are publicly available at https://github.com/mbzuai-nlp/bactrian-x