CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdom
This addresses the challenge of efficient instruction data selection for model training in AI, offering a novel method that outperforms existing approaches, though it is incremental in the context of synthetic data optimization.
The paper tackles the problem of selecting synthetic instruction data for distilling large language models into smaller ones by proposing CrowdSelect, a metric that uses multi-LLM wisdom and clustering to improve selection, achieving state-of-the-art performance with improvements of 4.81% on Arena-Hard and 11.1% on MT-bench for a specific model.
Distilling advanced Large Language Models' instruction-following capabilities into smaller models using a selected subset has become a mainstream approach in model training. While existing synthetic instruction data selection strategies rely mainly on single-dimensional signals (i.e., reward scores, model perplexity), they fail to capture the complexity of instruction-following across diverse fields. Therefore, we investigate more diverse signals to capture comprehensive instruction-response pair characteristics and propose three foundational metrics that leverage Multi-LLM wisdom, informed by (1) diverse LLM responses and (2) reward model assessment. Building upon base metrics, we propose CrowdSelect, an integrated metric incorporating a clustering-based approach to maintain response diversity. Our comprehensive experiments demonstrate that our foundation metrics consistently improve performance across 4 base models on MT-bench and Arena-Hard. CrowdSelect, efficiently incorporating all metrics, achieves state-of-the-art performance in both Full and LoRA fine-tuning, showing improvements of 4.81% on Arena-Hard and 11.1% on MT-bench with Llama-3.2-3b-instruct. We hope our findings will bring valuable insights for future research in this direction. Code are available at https://github.com/listentm/crowdselect.