Universal Reasoning Model
This work addresses the need for better reasoning models in AI, particularly for tasks like ARC-AGI, but it is incremental as it builds on existing universal transformers with specific enhancements.
The paper tackles the problem of understanding and improving universal transformers for complex reasoning tasks like ARC-AGI, finding that performance gains come from recurrent inductive bias and nonlinear components, and proposes the Universal Reasoning Model (URM) which achieves state-of-the-art results of 53.8% pass@1 on ARC-AGI 1 and 16.0% pass@1 on ARC-AGI 2.
Universal transformers (UTs) have been widely used for complex reasoning tasks such as ARC-AGI and Sudoku, yet the specific sources of their performance gains remain underexplored. In this work, we systematically analyze UTs variants and show that improvements on ARC-AGI primarily arise from the recurrent inductive bias and strong nonlinear components of Transformer, rather than from elaborate architectural designs. Motivated by this finding, we propose the Universal Reasoning Model (URM), which enhances the UT with short convolution and truncated backpropagation. Our approach substantially improves reasoning performance, achieving state-of-the-art 53.8% pass@1 on ARC-AGI 1 and 16.0% pass@1 on ARC-AGI 2. Our code is avaliable at https://github.com/UbiquantAI/URM.