One-Shot Learning as Instruction Data Prospector for Large Language Models
This addresses data quality issues in instruction tuning for LLM developers, offering an efficient alternative to scaling without clear strategies, though it is incremental as it builds on existing tuning practices.
The paper tackles the problem of noise in instruction tuning for large language models by introducing Nuggets, a method that uses one-shot learning to select high-quality instruction data, resulting in substantial performance improvements on benchmarks like MT-Bench and Alpaca-Eval with only the top 1% of curated examples.
Contemporary practices in instruction tuning often hinge on enlarging data scaling without a clear strategy for ensuring data quality, inadvertently introducing noise that may compromise model performance. To address this challenge, we introduce \textsc{Nuggets}, a novel and efficient methodology that leverages one-shot learning to discern and select high-quality instruction data from extensive datasets. \textsc{Nuggets} assesses the potential of individual instruction examples to act as effective one-shot learning instances, thereby identifying those that can significantly improve performance across diverse tasks. \textsc{Nuggets} utilizes a scoring system based on the impact of candidate examples on the perplexity of a diverse anchor set, facilitating the selection of the most advantageous data for instruction tuning. Through comprehensive evaluations on two benchmarks, including MT-Bench and Alpaca-Eval, we show that instruction tuning with the top 1\% of examples curated by \textsc{Nuggets} substantially outperforms conventional methods employing the entire dataset.