Tiny Recursive Reasoning with Mamba-2 Attention Hybrid
This work addresses the problem of efficient operator selection for recursive reasoning models, showing incremental improvements in candidate coverage for abstract reasoning tasks.
The paper investigated whether replacing Transformer blocks with Mamba-2 hybrid operators in a tiny recursive reasoning model (TRM) preserves reasoning capability, finding that the hybrid improved pass@2 by +2.0% (45.88% vs 43.88%) and pass@100 by +4.75% on ARC-AGI-1 while maintaining parameter parity.
Recent work on recursive reasoning models like TRM demonstrates that tiny networks (7M parameters) can achieve strong performance on abstract reasoning tasks through latent recursion -- iterative refinement in hidden representation space without emitting intermediate tokens. This raises a natural question about operator choice: Mamba-2's state space recurrence is itself a form of iterative refinement, making it a natural candidate for recursive reasoning -- but does introducing Mamba-2 into the recursive scaffold preserve reasoning capability? We investigate this by replacing the Transformer blocks in TRM with Mamba-2 hybrid operators while maintaining parameter parity (6.83M vs 6.86M parameters). On ARC-AGI-1, we find that the hybrid improves pass@2 (the official metric) by +2.0\% (45.88\% vs 43.88\%) and consistently outperforms at higher K values (+4.75\% at pass@100), whilst maintaining pass@1 parity. This suggests improved candidate coverage -- the model generates correct solutions more reliably -- with similar top-1 selection. Our results validate that Mamba-2 hybrid operators preserve reasoning capability within the recursive scaffold, establishing SSM-based operators as viable candidates in the recursive operator design space and taking a first step towards understanding the best mixing strategies for recursive reasoning.