Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL
This addresses the problem of enabling LLMs to continually improve beyond fixed training budgets for researchers and practitioners in AI, though it appears incremental as it builds on existing RL and LLM capabilities.
The paper tackles the problem of limited extrapolation in LLMs during test-time distribution shifts by introducing Reasoning Cache (RC), an iterative decoding algorithm that replaces standard autoregressive decoding. Training a 4B model with RC improved performance on HMMT 2025 from 40% to nearly 70% using a 16k-token training budget and 0.5m tokens at test time.
Large Language Models (LLMs) that can continually improve beyond their training budgets are able to solve increasingly difficult problems by adapting at test time, a property we refer to as extrapolation. However, standard reinforcement learning (RL) operates over fixed problem distributions and training budgets, which limits extrapolation amidst distribution shift at test time. To address this, we introduce RC, an iterative decoding algorithm that replaces standard autoregressive decoding during both training and inference. RC exploits an asymmetry between the response generation and summarization capabilities of LLMs to construct reasoning chains that consistently improve across iterations. Models trained to use RC can extrapolate and continually improve over reasoning horizons more than an order of magnitude longer than those seen during training. Empirically, training a 4B model with RC using a 16k-token training budget improves performance on HMMT 2025 from 40% to nearly 70% with 0.5m tokens at test time, outperforming both comparably sized models and many larger reasoning LLMs. Finally, we also show that models trained with RC can more effectively leverage existing scaffolds to further scale test-time performance, due to the improved summary-conditioned generation abilities learned through training.