Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RL
This work addresses the challenge of dynamic temperature control in LLMs for tasks like mathematical reasoning, offering an incremental improvement over existing methods by integrating temperature selection with task rewards.
The paper tackles the problem of optimizing sampling temperature in large language models (LLMs) for better exploration-exploitation trade-offs during generation, proposing a hierarchical reinforcement learning framework that learns temperature policies from internal states, resulting in improved performance on mathematical reasoning benchmarks compared to fixed or heuristic baselines.
Reinforcement Learning from Verifiable Rewards (RLVR) trains large language models (LLMs) from sampled trajectories, making decoding strategy a core component of learning rather than a purely inference-time choice. Sampling temperature directly controls the exploration--exploitation trade-off by modulating policy entropy, yet existing methods rely on static values or heuristic adaptations that are decoupled from task-level rewards. We propose Introspective LLM, a hierarchical reinforcement learning framework that learns to control sampling temperature during generation. At each decoding step, the model selects a temperature based on its hidden state and samples the next token from the resulting distribution. Temperature and token policies are jointly optimized from downstream rewards using a coordinate ascent scheme. Experiments on mathematical reasoning benchmarks show that learned temperature policies outperform fixed and heuristic baselines, while exhibiting interpretable exploration behaviors aligned with reasoning uncertainty.