Latent Poincaré Shaping for Agentic Reinforcement Learning
This work addresses the challenge of enhancing reasoning and search capabilities in LLM agents for mathematical problem-solving, representing a novel approach with significant performance gains.
The paper tackles the problem of training AlphaZero-like LLM agents by proposing LaPha, a method that uses a Poincaré latent space to improve search efficiency and accuracy, achieving improvements from 66.0% to 88.2% on MATH-500 and up to 60.0% accuracy on AIME'24.
We propose LaPha, a method for training AlphaZero-like LLM agents in a Poincaré latent space. Under LaPha, the search process can be visualized as a tree rooted at the prompt and growing outward from the origin toward the boundary of the Poincaré ball, where negative curvature provides exponentially increasing capacity with radius. Using hyperbolic geodesic distance to rule-verified correctness, we define a node potential and assign dense process rewards by potential differences. We further attach a lightweight value head on the same shared latent space, enabling self-guided test-time scaling with almost no additional overhead. On MATH-500, LaPha improves Qwen2.5-Math-1.5B from 66.0% to 88.2%. With value-head-guided search, LaPha-1.5B reaches 56.7% accuracy on AIME'24, and LaPha-7B further achieves 60.0% on AIME'24 and 53.3% on AIME'25.