Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca
This provides a cost-efficient guide for expanding language support in AI applications like chat assistants, though it is incremental in nature.
The study compared monolingual and multilingual instruction tuning for large language models, finding that multilingual tuning performs as well or better than per-language tuning under a fixed computation budget, with downsampled data also proving effective and robust.
Foundational large language models (LLMs) can be instruction-tuned to perform open-domain question answering, facilitating applications like chat assistants. While such efforts are often carried out in a single language, we empirically analyze cost-efficient strategies for multilingual scenarios. Our study employs the Alpaca dataset and machine translations of it to form multilingual data, which is then used to tune LLMs through either low-rank adaptation or full-parameter training. Under a controlled computation budget, comparisons show that multilingual tuning is on par or better than tuning a model for each language. Furthermore, multilingual tuning with downsampled data can be as powerful and more robust. Our findings serve as a guide for expanding language support through instruction tuning.