Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimization
This work addresses a foundational issue in AI by expanding reasoning capacity in LLMs, which is crucial for advancing complex problem-solving in domains like mathematics, though it appears incremental as it builds on existing RLVR methods.
The paper tackles the problem of whether reinforcement learning genuinely expands reasoning capacity in large language models or merely aligns existing capabilities, by proposing Manifold-Reshaping Policy Optimization (MRPO) to fundamentally restructure the inference space, achieving state-of-the-art performance on mathematical tasks and significantly outperforming larger models like Qwen3-32B.
Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). However, recent studies question whether RL genuinely expands reasoning capacity or merely aligns existing latent capabilities, arguing that exploration remains confined within the pre-trained model's low-rank bias manifold. In this work, we challenge this accessibility boundary hypothesis by demonstrating that the latent reasoning space can be fundamentally expanded through targeted geometric interventions. We propose Manifold-Reshaping Policy Optimization (MRPO), a geometric framework designed to fundamentally restructure the inference space of LLMs. MRPO operates in two stages: first, we employ Spectral Orthogonal Exploration (SOE) to eject the policy initialization into the null space of the bias manifold; second, we integrate an Effective Rank regularization term into the policy optimization objective. This approach incentivizes the discovery and maintenance of high-dimensional reasoning trajectories against the entropy-reducing tendency of standard RL. Empirically, our 4B-parameter method achieves state-of-the-art performance on mathematical tasks, significantly outperforming larger models (e.g., Qwen3-32B) and expanding the capability boundary beyond standard GRPO. Our code is available at https://anonymous.4open.science/r/MRPO-D57B/