Learning to Refine Hidden States for Reliable LLM Reasoning
For LLM practitioners, ReLAR offers a more efficient and stable alternative to explicit chain-of-thought reasoning, addressing error propagation in multi-step tasks.
ReLAR improves LLM reasoning accuracy and stability by iteratively refining hidden states before decoding, using reinforcement learning to control refinement depth and direction. On medical, math, multi-hop, and open-ended benchmarks, it achieves higher quality with lower inference cost than chain-of-thought methods.
Large language models show strong reasoning ability, but their internal reasoning process can remain unstable in complex multi-step settings, where early hidden-state errors may propagate to incorrect predictions. We propose ReLAR, a reinforcement-guided latent refinement framework that iteratively updates hidden representations before decoding. ReLAR maintains a compact latent reasoning state and uses learned depth and action controllers to adaptively determine both the number and direction of refinement steps. The controllers are trained with a policy gradient objective based on step-wise likelihood improvement, enabling efficient input-dependent reasoning without explicit chain-of-thought generation. Experiments on medical, mathematical, multi-hop reasoning, and open-ended generation benchmarks show that ReLAR improves accuracy, generation quality, and reasoning stability with substantially lower inference overhead than explicit reasoning baselines.