Tool Building as a Path to "Superintelligence"
This addresses the challenge of evaluating superintelligence pathways in LLMs, though it appears incremental as it builds on the existing Diligent Learner framework with a new benchmark.
The paper tackles the problem of measuring LLMs' ability to achieve superintelligence via test-time search by designing a benchmark for logical out-of-distribution inference tasks involving GF(2) circuit reconstruction, finding that frontier models show partial robustness while small LLMs decline superlinearly with depth, and that precise tool calls are critical for success.
The Diligent Learner framework suggests LLMs can achieve superintelligence via test-time search, provided a sufficient step-success probability $γ$. In this work, we design a benchmark to measure $γ$ on logical out-of-distribution inference. We construct a class of tasks involving GF(2) circuit reconstruction that grow more difficult with each reasoning step, and that are, from an information-theoretic standpoint, impossible to reliably solve unless the LLM carefully integrates all of the information provided. Our analysis demonstrates that while the $γ$ value for small LLMs declines superlinearly as depth increases, frontier models exhibit partial robustness on this task. Furthermore, we find that successful reasoning at scale is contingent upon precise tool calls, identifying tool design as a critical capability for LLMs to achieve general superintelligence through the Diligent Learner framework.