ToolExpander: Extending the Frontiers of Tool-Using Reinforcement Learning to Weak LLMs
This addresses the problem of training instability and poor performance in resource-constrained LLMs for tool-using reinforcement learning, representing an incremental improvement over existing methods.
The paper tackled the challenge of training small-scale Large Language Models (LLMs) with Group Relative Policy Optimization (GRPO), which often leads to inaccurate responses and mid-training collapse, by proposing ToolExpander, a framework that improved tool-using capabilities, enhancing training stability and performance in weaker models.
Training Large Language Models (LLMs) with Group Relative Policy Optimization (GRPO) encounters a significant challenge: models often fail to produce accurate responses, particularly in small-scale architectures. This limitation not only diminishes performance improvements and undermines the potential of GRPO but also frequently leads to mid-training collapse, adversely affecting stability and final efficacy. To address these issues, we propose ToolExpander, a novel framework that advances tool-oriented reinforcement learning for resource-constrained LLMs through two key innovations:(1) Dynamic Multi-Round Hard Sampling, which dynamically substitutes challenging samples(those without correct outputs over 10 rollouts) with high-quality few-shot demonstrations during training, coupled with an exponential learning rate decay strategy to mitigate oscillations;(2) Self-Exemplifying Thinking, an enhanced GRPO framework that eliminates KL divergence and incorporates adjusted clipping coefficients, encouraging models to autonomously generate and analyze few-shot examples via a minimal additional reward (0.01).Experimental results demonstrate that ToolExpander significantly enhances tool-using capabilities in LLMs, especially in weaker small-scale models, improving both training stability and overall performance.