Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning
This addresses the challenge of efficiently identifying high-value training data for fine-tuning LLMs, which is incremental as it builds on existing data selection methods but introduces a novel theoretical grounding.
The paper tackles the problem of noisy or off-target examples diluting supervision in fine-tuning large language models by proposing a resource-efficient framework for data selection and reweighting based on in-context learning, resulting in consistent improvements in model alignment across various methods and datasets with minimal overhead.
Fine-tuning large pretrained language models is a common approach for aligning them with human preferences, but noisy or off-target examples can dilute supervision. While small, well-chosen datasets often match the performance of much larger ones, systematic and efficient ways to identify high-value training data remain underexplored. Many current methods rely on heuristics or expensive retraining. We present a theoretically grounded, resource-efficient framework for data selection and reweighting. At its core is an In-Context Approximation (ICA) that estimates the holdout loss a model would incur after training on a candidate example by conditioning on a small, curated holdout set in context. ICA requires no reference model and no additional finetuning. Under a local linearization, ICA is equivalent to a first-order update toward the holdout optimum, motivating its use as a proxy for data value. We derive per-example weights from ICA scores, dynamically reweighting gradient updates as model parameters evolve. Across SFT, DPO, and SimPO, and over diverse backbones and datasets, ICA-based reweighting consistently improves model alignment with minimal overhead. We analyze sensitivity to score update frequency and the choice of $k$ holdout examples for in-context demonstrations, and note limitations for rapidly drifting on-policy updates, highlighting directions for future work. Code and prompts will be released.