From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models
This work addresses the problem of limited interactive capabilities for large language models in Japanese, providing a dataset and tuning methods, but it is incremental as it builds on existing techniques and datasets.
The authors tackled the lack of instruction tuning datasets for non-English languages by constructing a Japanese instruction dataset and applying it to Japanese and English large language models using Low-Rank Adaptation tuning. They confirmed the effectiveness of Japanese instruction datasets and showed that even small models improve in downstream tasks through instruction tuning.
Instruction tuning is essential for large language models (LLMs) to become interactive. While many instruction tuning datasets exist in English, there is a noticeable lack in other languages. Also, their effectiveness has not been well verified in non-English languages. We construct a Japanese instruction dataset by expanding and filtering existing datasets and apply the dataset to a Japanese pre-trained base model. We performed Low-Rank Adaptation (LoRA) tuning on both Japanese and English existing models using our instruction dataset. We evaluated these models from both quantitative and qualitative perspectives. As a result, the effectiveness of Japanese instruction datasets is confirmed. The results also indicate that even with relatively small LLMs, performances in downstream tasks would be improved through instruction tuning. Our instruction dataset, tuned models, and implementation are publicly available online.