Beyond Sample-Level Feedback: Using Reference-Level Feedback to Guide Data Synthesis
This addresses the problem of scalable, high-quality data synthesis for LLM instruction-tuning, offering a novel approach that outperforms existing methods.
The paper tackles the quality ceiling in synthetic data generation for instruction-tuning LLMs by introducing Reference-Level Feedback, which extracts characteristics from curated references to guide synthesis, resulting in a 43.96% win-rate on AlpacaEval 2.0 with fine-tuned models.
High-quality instruction-tuning data is crucial for developing Large Language Models (LLMs) that can effectively navigate real-world tasks and follow human instructions. While synthetic data generation offers a scalable approach for creating such datasets, it imposes a quality ceiling where models trained on the data cannot outperform the LLM generating it. To overcome this limitation, we introduce Reference-Level Feedback, a paradigm that extracts desirable characteristics from carefully curated reference samples to guide the synthesis of higher-quality instruction-response pairs. Using this approach, we synthesize REFED, a dataset of 10K instruction-response pairs. Fine-tuning Llama-3.1-8B-Instruct and Mistral-7B-Instruct on REFED demonstrate state-of-the-art performance among similarly sized models, notably reaching a 43.96\% length-controlled win-rate on AlpacaEval 2.0. Extensive experiments demonstrate that Reference-Level Feedback consistently outperforms traditional sample-level feedback methods, generalizes across model architectures, and produces high-quality and diverse data at low cost.