DavIR: Data Selection via Implicit Reward for Large Language Models
This addresses the challenge of efficient data selection for LLM fine-tuning, offering a way to reduce training costs and improve model performance, though it is incremental as it builds on existing methods like DPO.
The paper tackles the problem of selecting high-quality data for fine-tuning large language models by introducing DavIR, a method that uses implicit reward to quantify data learnability, and shows that using only 6% of the Alpaca dataset selected with DavIR achieves superior performance compared to full-dataset training on models like LLaMA and Gemma, and improves alignment performance by 8% on AlpacaEval.
We introduce DavIR, a model-based data selection method for post-training Large Language Models. DavIR generalizes Reducible Holdout Loss to core-set selection problem of causal language modeling, and quantifies the learnability of a given datum with respect to a pre-trained LLM based on relative reduction in loss during fine-tuning, a metric we show to be closely related to the implicit reward model described in Direct Preference Optimization (DPO). We show that 6% of Alpaca dataset selected with DavIR can steer both the LLaMA and Gemma model family to produce superior performance compared to the same models trained on the full 52K dataset. We also show that Alpaca dataset compressed with DavIR can be combined with GSM8K dataset to effectively balance open-domain freeform QA and mathematical reasoning capabilities. Finally, we apply the DavIR objective to DPO and develop a normalized DavIR-DPO objective which improves alignment performance of Zephyr-7B-SFT model by 8% (relative) on AlpacaEval, compared against training on vanilla DPO objective.