Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Models
For researchers improving LLM reasoning via RL, this work highlights the importance of incorporating task structure and type-awareness into problem sampling, moving beyond difficulty-based curricula.
The paper frames problem sampling in LLM reinforcement learning as a manifold-structured bandit problem and introduces Bayesian Manifold Curriculum (BMC), a structure-aware framework that organizes problems into a hierarchical task tree. Empirical results show that different sampling strategies induce tradeoffs between productivity, diversity, and utility, and that prioritizing difficulty alone is insufficient for strong downstream performance.
Reinforcement learning (RL) is a central approach for improving reasoning capabilities in large language models (LLMs), where training efficiency depends critically on how problems are sampled during optimization. Existing adaptive curriculum learning methods typically prioritize prompts of intermediate difficulty, treating problem selection as a standard bandit problem with independent arms and overlooking the structured, heterogeneous nature of the task space. In this work, we frame problem sampling as a manifold-structured bandit problem with endogenous non-stationarity: problems are related through the model's latent representation space, and sampling decisions can steer how learning signals evolve across that space. To operationalize this perspective, we introduce Bayesian Manifold Curriculum (BMC), a structure-aware framework that organizes problems into a hierarchical task tree and applies Bayesian learning to guide sampling. Empirically, we find that different sampling strategies induce non-trivial tradeoffs between productivity (learning signal), diversity (coverage of the task manifold), and utility (evaluation relevance). These results show that prioritizing difficulty alone is insufficient for strong downstream performance, highlighting the importance of incorporating structure and type-awareness into problem sampling.