Think, But Don't Overthink: Reproducing Recursive Language Models
This work addresses the problem of optimizing recursion depth in Recursive Language Models for researchers and developers working with Large Language Models, providing insights into the limitations of deeper recursion.
The study investigates the impact of scaling recursion depth in Recursive Language Models, finding that deeper recursion can cause models to 'overthink' and degrade performance, with execution time increasing from 3.6s to 344.5s. The results show that depth-1 RLMs boost accuracy on complex reasoning tasks, but deeper recursion paradoxically degrades performance.
This project reproduces and extends the recently proposed ``Recursive Language Models'' (RLMs) framework by Zhang et al. (2026). This framework enables Large Language Models (LLMs) to process near-infinite contexts by offloading the prompt into an external REPL environment. While the original paper relies on a default recursion depth of 1 and suggests deeper recursion as a future direction, this study specifically investigates the impact of scaling the recursion depth. Using state-of-the-art open-source agentic models (DeepSeek v3.2 and Kimi K2), I evaluated pure LLM, RLM (depth=1), and RLM (depth=2) on the S-NIAH and OOLONG benchmarks. The findings reveal a compelling phenomenon: Deeper recursion causes models to ``overthink''. While depth-1 RLMs effectively boost accuracy on complex reasoning tasks, applying deeper recursion (depth=2) or using RLMs on simple retrieval tasks paradoxically degrades performance and exponentially inflates execution time (e.g., from 3.6s to 344.5s) and token costs. Code and data are available at: https://github.com/drbillwang/rlm-reproduction