Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation
This work addresses the need for better Chinese conversational models by providing insights for open-source development, though it is incremental as it builds on existing methods like LLaMA.
The study investigated how training data factors like quantity, quality, and linguistic distribution affect the performance of instruction-following language models for Chinese, using evaluations on 1,000 samples across nine scenarios and extending LLaMA's vocabulary with secondary pre-training on 3.4B Chinese words.
Recently, significant public efforts have been directed towards developing low-cost models with capabilities akin to ChatGPT, thereby fostering the growth of open-source conversational models. However, there remains a scarcity of comprehensive and in-depth evaluations of these models' performance. In this study, we examine the influence of training data factors, including quantity, quality, and linguistic distribution, on model performance. Our analysis is grounded in several publicly accessible, high-quality instruction datasets, as well as our own Chinese multi-turn conversations. We assess various models using a evaluation set of 1,000 samples, encompassing nine real-world scenarios. Our goal is to supplement manual evaluations with quantitative analyses, offering valuable insights for the continued advancement of open-source chat models. Furthermore, to enhance the performance and training and inference efficiency of models in the Chinese domain, we extend the vocabulary of LLaMA - the model with the closest open-source performance to proprietary language models like GPT-3 - and conduct secondary pre-training on 3.4B Chinese words. We make our model, data, as well as code publicly available.