Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
For researchers working on deep reasoning models, this work provides a method to stabilize looped architectures and adapt computation dynamically, though it is incremental over existing looped Transformer approaches.
The paper addresses signal propagation issues in looped Transformer architectures for compositional reasoning tasks. By using pre-norm layers and residual scaling, the proposed FPRM model achieves fixed-point convergence as a halting mechanism, adapting compute to task difficulty and showing effectiveness on Sudoku, Maze, state-tracking, and ARC-AGI benchmarks.
Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The number of effective layers reached by looping determines the quality of the solution these models find. Like deep architectures, looped architectures are prone to a signal propagation problem induced by depth as the halting decision is postponed. In this paper, we address this signal propagation issue using pre-norm layers and residual scaling. Building on these architectural modifications, we propose FPRM, a Transformer-based Fixed-Point Reasoning Model that uses fixed-point convergence as an end-to-end halting mechanism in a looped architecture. We show that fixed-point halting allows FPRM to adapt its compute to task difficulty. FPRM is effective on common reasoning benchmarks, namely Sudoku, Maze, state-tracking, and ARC-AGI.