Tiny Autoregressive Recursive Models
This research investigates the potential of a specific refinement mechanism for autoregressive models, providing a negative result that cautions against further investment in the Autoregressive TRM architecture for the machine learning research community.
This paper explores the application of Tiny Recursive Models (TRMs) to autoregressive tasks, proposing the Autoregressive TRM. Through controlled experiments on character-level algorithmic tasks, the authors found that while some two-level refinement baselines showed strong performance, the full Autoregressive TRM architecture did not yield reliable performance gains.
Tiny Recursive Models (TRMs) have recently demonstrated remarkable performance on ARC-AGI, showing that very small models can compete against large foundation models through a two-step refinement mechanism that updates an internal reasoning state $z$ and the predicted output $y$. Naturally, such refinement is of interest for any predictor; it is therefore natural to wonder whether the TRM mechanism could be effectively re-adopted in autoregressive models. However, TRMs cannot be simply compared to standard models because they lack causal predictive structures and contain persistent latent states that make it difficult to isolate specific performance gains. In this paper, we propose the Autoregressive TRM and evaluate it on small autoregressive tasks. To understand its efficacy, we propose a suite of models that gradually transform a standard Transformer to a Tiny Autoregressive Recursive Model in a controlled setting that fixes the block design, token stream, and next-token objective. Across compute-matched experiments on character-level algorithmic tasks, we surprisingly find that there are some two-level refinement baselines that show strong performance. Contrary to expectations, we find no reliable performance gains from the full Autoregressive TRM architecture. These results offer potential promise for two-step refinement mechanisms more broadly but caution against investing in the autoregressive TRM-specific model as a fruitful research direction.