TACOS: Open Tagging and Comparative Scoring for Instruction Fine-Tuning Data Selection
This addresses the challenge of efficient and effective data selection for aligning LLMs with human preferences, offering a novel method that improves over existing heuristics and singleton evaluations.
The paper tackles the problem of selecting a small yet representative subset for instruction fine-tuning of large language models by introducing TACOS, which integrates open tagging and comparative scoring to improve data diversity and quality evaluation, resulting in superior performance on benchmarks like MT-Bench and ranking 1st among LLaMA2-7B-based models on AlpacaEval 2.0.
Instruction Fine-Tuning (IFT) is crucial for aligning large language models (LLMs) with human preferences, and selecting a small yet representative subset from massive data significantly facilitates IFT in terms of both efficiency and effectiveness. Nevertheless, existing approaches suffer from two limitations: the use of simple heuristics restricts data diversity, while the singleton data quality evaluation accounts for inconsistent criteria between independent samples. To address the issues, we present TACOS, an innovative method that integrates Open Tagging and Comparative Scoring for IFT data selection. To capture data diversity, we leverage LLMs to assign open-domain tags to human queries, followed by a normalization stage to denoise the open tags and enable efficient clustering. Additionally, we suggest a comparative scoring method that allows the relative quality evaluation of samples within a cluster, avoiding inconsistent criteria seen in singleton-based evaluations. Extensive experiments across diverse datasets and LLM architectures demonstrate that TACOS outperforms existing approaches by a large margin. Notably, it achieves superior instruction-following performance on MT-Bench and ranks 1st among LLaMA2-7B-Based models on AlpacaEval 2.0, illustrating its efficacy for IFT data selection.