Token-Efficient RL for LLM Reasoning
This work addresses the challenge of optimizing LLM reasoning for resource-limited settings, offering incremental improvements in token-efficient RL methods.
The authors tackled the problem of efficient reinforcement learning for reasoning in large language models under memory and compute constraints, achieving an accuracy increase from 46% to over 70% on the SVAMP benchmark with Qwen2-1.5B.
We propose reinforcement learning (RL) strategies tailored for reasoning in large language models (LLMs) under strict memory and compute limits, with a particular focus on compatibility with LoRA fine-tuning. Building on early policy gradient methods with baseline subtraction, we design critic-free methods that operate on a small, informative subset of output tokens to reduce memory usage and stabilize training. We introduce S-GRPO, a stochastic variant of Group Relative Policy Optimization, and T-SPMO, a token-level prefix matching approach for fine-grained credit assignment. Applied to Qwen2-1.5B, our methods raise accuracy on the SVAMP benchmark from 46% to over 70% and show strong performance on multi-digit multiplication. Surprisingly, full-token GRPO under LoRA fails to improve over the base model, suggesting that selective token-level optimization may act as an implicit regularizer in low-parameter training regimes.