Efficient Code LLM Training via Distribution-Consistent and Diversity-Aware Data Selection
This addresses training efficiency for code LLMs, offering a domain-specific incremental improvement.
The paper tackles inefficient training of code large language models by proposing a data selection method that ensures distribution consistency and diversity, achieving gains of 2.4% on HumanEval and 2.3% on MBPP with only 10K samples compared to a 92K baseline.
Recent advancements in large language models (LLMs) have significantly improved code generation and program comprehension, accelerating the evolution of software engineering. Current methods primarily enhance model performance by leveraging vast amounts of data, focusing on data quantity while often overlooking data quality, thereby reducing training efficiency. To address this, we introduce an approach that utilizes a parametric model for code data selection, aimed at improving both training efficiency and model performance. Our method optimizes the parametric model to ensure distribution consistency and diversity within the selected subset, guaranteeing high-quality data. Experimental results demonstrate that using only 10K samples, our method achieves gains of 2.4% (HumanEval) and 2.3% (MBPP) over 92K full-sampled baseline, outperforming other sampling approaches in both performance and efficiency. This underscores that our method effectively boosts model performance while significantly reducing computational costs.