HyperGuide: Hyperbolic Guidance for Efficient Multi-Step Reasoning in Large Language Models
This work addresses the efficiency-accuracy tradeoff in multi-step reasoning for LLMs, offering a lightweight method to improve reasoning without heavy computation.
HyperGuide introduces a hyperbolic geometric signal to guide multi-step reasoning in LLMs, achieving consistent accuracy gains with larger improvements on deeper reasoning chains, while remaining more efficient than tree-search methods.
Multi-step reasoning remains a central challenge for large language models: single-pass generation is efficient but lacks accuracy; tree-search methods explore multiple paths but are computation-heavy. We address this gap by distilling reasoning progress into a hyperbolic geometric signal that guides step-by-step generation. Our approach is motivated by a structural observation: in combinatorial reasoning trees, solution-bearing states are few while dead ends are exponentially numerous. The hyperbolic space matches this asymmetry, with compact volume near the origin and exponentially expanding capacity toward the boundary, so that distance-to-origin naturally encodes solution proximity while angular separation distinguishes branches requiring different next operations. We train a lightweight head to project LLM hidden states into this space, then fine-tune a low-rank adapter interactively on its own reasoning attempts to act on the injected signal. Across multiple benchmarks, the geometric signal yields consistent gains, with larger improvements on deeper reasoning chains. Our code is publicly available at https://github.com/yuyuliu11037/HyperGuide.