ADAPT: Learning Task Mixtures for Budget-Constrained Instruction Tuning
This addresses the challenge of efficient resource allocation in instruction tuning for LLMs, offering a method to improve performance under budget constraints, though it is incremental as it builds on existing meta-learning and curriculum learning approaches.
The paper tackled the problem of optimizing task sampling proportions for multi-task instruction tuning under a token budget, proposing ADAPT, a meta-learning algorithm that learns an adaptive curriculum. The result showed that ADAPT matched or slightly improved average downstream performance on 11 out-of-domain benchmarks compared to the best static mixture, while using fewer effective training tokens and reallocating budget toward harder tasks.
We propose ADAPT, a meta-learning algorithm that \emph{learns} task sampling proportions under an explicit token budget for multi-task instruction tuning. Instead of fixing task weights by hand, \adapt{} maintains a continuous distribution over tasks and updates it via meta-gradients of a smooth worst-case validation objective, inducing an adaptive curriculum that allocates more tokens to useful tasks while avoiding collapse. We instantiate ADAPT on three $\sim$1B-parameter open-weight LLMs (Gemma-3-1B, LLaMA-3.2-1B, Qwen-0.6B), training on 20 Natural Instructions task types under budgets of $1\%$, $5\%$, and $10\%$ of the available supervised tokens, and compare against strong supervised fine-tuning baselines with uniform and size-proportional mixing. We conduct evaluations on 11 out-of-domain benchmarks spanning reasoning, reading comprehension, code generation, and instruction following, we find that ADAPT matches or slightly improves average downstream performance relative to the best static mixture, while using fewer effective training tokens and reallocating budget toward harder, benchmark-aligned tasks.