The Deterministic Horizon: When Extended Reasoning Fails and Tool Delegation Becomes Necessary
Provides principled guidance for when pure neural reasoning should yield to hybrid approaches in agentic systems, addressing a critical bottleneck for reliable reasoning in LLMs.
The paper identifies a fundamental architectural limitation in decoder-only attention models for deterministic state-tracking tasks, proving that extended chain-of-thought reasoning degrades beyond a deterministic horizon (d* ∈ [19, 31]), where tool delegation becomes necessary. Across 12 models and 8 domains, tool-integrated reasoning achieves 86-94% accuracy versus 24-42% for neural chain-of-thought.
Extended chain-of-thought reasoning can degrade performance on deterministic state-tracking tasks, not due to preference biases, but limits rooted in the information-theoretic capacity of decoder-only attention. We establish: (1) an Attention Bottleneck Theorem with a complementary achievability construction, bounding state-tracking capacity as $O(H \cdot \log(L/H) \cdot \sqrt{d_h})$; (2) a context-dependent error model yielding super-exponential accuracy decay; (3) the State-Space Jaccard metric distinguishing capability from preference failures; (4) a Deterministic Horizon $d^* \in [19, 31]$ beyond which tool delegation becomes necessary. Across 12 models and 8 task domains (including SWE-Bench, WebArena, and SQL-Multi), tool-integrated reasoning consistently outperforms neural chain-of-thought; on the primary model suite it reaches 86-94% accuracy versus 24-42% for neural chain-of-thought. Fine-tuning on optimal-length traces yields $<$5% improvement, confirming an architectural ceiling, and high cross-model correlation ($r = 0.81$-$0.91$) indicates these failures are architectural rather than training-specific. Our results provide principled guidance for when pure neural reasoning should yield to hybrid approaches in agentic systems.