How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Data
This addresses the challenge of identifying genuinely high-quality data for code LLMs, which is crucial for developers and researchers building reliable code generation tools, though it is incremental as it builds on existing tuning methods.
The paper tackles the problem of data leakage in code instruction tuning datasets, which leads to inflated performance on benchmarks like HumanEval but poor results on others like LiveCodeBench, and proposes a data pruning strategy based on instruction complexity, response quality, and diversity, resulting in XCoder models that achieve state-of-the-art performance with fewer training data.
Recently, there has been a growing interest in studying how to construct better code instruction tuning data. However, we observe Code models trained with these datasets exhibit high performance on HumanEval but perform worse on other benchmarks such as LiveCodeBench. Upon further investigation, we find that many datasets suffer from severe data leakage. After cleaning up most of the leaked data, some well-known high-quality datasets perform poorly. This discovery reveals a new challenge: identifying which dataset genuinely qualify as high-quality code instruction data. To address this, we propose an efficient code data pruning strategy for selecting good samples. Our approach is based on three dimensions: instruction complexity, response quality, and instruction diversity. Based on our selected data, we present XCoder, a family of models finetuned from LLaMA3. Our experiments show XCoder achieves new state-of-the-art performance using fewer training data, which verify the effectiveness of our data strategy. Moreover, we perform a comprehensive analysis on the data composition and find existing code datasets have different characteristics according to their construction methods, which provide new insights for future code LLMs. Our models and dataset are released in https://github.com/banksy23/XCoder