Efficient RL Training for Reasoning Models via Length-Aware Optimization
This addresses efficiency issues in reasoning models for AI applications, though it appears incremental as it builds on existing RL training methods.
The paper tackles the problem of long reasoning paths in large reasoning models by proposing three reward designs integrated directly into reinforcement learning training, achieving a 40% reduction in response length with a 14% performance gain in logic reasoning and a 33% reduction while preserving performance in math problems.
Large reasoning models, such as OpenAI o1 or DeepSeek R1, have demonstrated remarkable performance on reasoning tasks but often incur a long reasoning path with significant memory and time costs. Existing methods primarily aim to shorten reasoning paths by introducing additional training data and stages. In this paper, we propose three critical reward designs integrated directly into the reinforcement learning process of large reasoning models, which reduce the response length without extra training stages. Experiments on four settings show that our method significantly decreases response length while maintaining or even improving performance. Specifically, in a logic reasoning setting, we achieve a 40% reduction in response length averaged by steps alongside a 14% gain in performance. For math problems, we reduce response length averaged by steps by 33% while preserving performance.