A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula
This provides theoretical insights for researchers and practitioners in AI working on self-improving models, though it is incremental as it builds on existing empirical methods.
The paper tackles the lack of theoretical foundation for iterative self-improvement in large language models by modeling it as maximum-likelihood fine-tuning and deriving finite-sample guarantees for expected reward, revealing a feedback loop that explains improvement and saturation. It proves conditions where easy-to-hard curricula outperform fixed task mixtures, validated through simulations and experiments on reasoning tasks.
Iterative self-improvement fine-tunes an autoregressive large language model (LLM) on reward-verified outputs generated by the LLM itself. In contrast to the empirical success of self-improvement, the theoretical foundation of this generative, iterative procedure in a practical, finite-sample setting remains limited. We make progress toward this goal by modeling each round of self-improvement as maximum-likelihood fine-tuning on a reward-filtered distribution and deriving finite-sample guarantees for the expected reward. Our analysis reveals an explicit feedback loop where better models accept more data per iteration, supporting sustained self-improvement while explaining eventual saturation of such improvement. Adopting a task-centric view by considering reasoning tasks with multiple difficulty levels, we further prove quantifiable conditions on model initialization, task difficulty, and sample budget where easy-to-hard curricula provably achieve better guarantees than training on fixed mixtures of tasks. Our analyses are validated via Monte-Carlo simulations and controlled experiments on graph-based reasoning tasks.