Boosting LLM via Learning from Data Iteratively and Selectively
This addresses data quality issues in instruction tuning for LLMs, though it is an incremental improvement over existing data selection methods.
The paper tackles the problem of noisy and duplicate data in instruction tuning by proposing IterIT, an iterative data selection method that measures sample quality through complexity and diversity scores. Experiments show consistent improvements over strong baselines across multiple instruction-tuning datasets and good generalization to domain-specific scenarios and different backbone models.
Datasets nowadays are generally constructed from multiple sources and using different synthetic techniques, making data de-noising and de-duplication crucial before being used for post-training. In this work, we propose to perform instruction tuning by iterative data selection (\ApproachName{}). We measure the quality of a sample from complexity and diversity simultaneously. Instead of calculating the complexity score once for all before fine-tuning, we highlight the importance of updating this model-specific score during fine-tuning to accurately accommodate the dynamic changes of the model. On the other hand, the diversity score is defined on top of the samples' responses under the consideration of their informativeness. IterIT integrates the strengths of both worlds by iteratively updating the complexity score for the top-ranked samples and greedily selecting the ones with the highest complexity-diversity score. Experiments on multiple instruction-tuning data demonstrate consistent improvements of IterIT over strong baselines. Moreover, our approach also generalizes well to domain-specific scenarios and different backbone models. All resources will be available at https://github.com/JiaQiSJTU/IterIT.