CodeTool: Enhancing Programmatic Tool Invocation of LLMs via Process Supervision
This addresses the challenge of reliable tool use in LLMs for developers and researchers, though it appears incremental as it builds on existing process supervision methods.
The paper tackles the problem of inefficient and unverifiable tool invocation in LLMs for complex tasks by proposing CodeTool, a framework that uses stepwise code generation with process supervision rewards, achieving superior performance on StableToolBench and RestBench-TMDB benchmarks.
Tool invocation significantly enhances the capabilities of Large Language Models (LLMs), yet challenges persist, particularly in complex task scenarios. Current methods, such as instruction-enhanced reasoning and supervised fine-tuning, often result in unnecessarily long reasoning paths and face difficulties in verifying the correctness of intermediate steps. In this paper, we propose CodeTool, a novel framework for stepwise code generation that improves LLM tool invocation by leveraging the concise and easily verifiable nature of code. CodeTool incorporates two distinct process rewards: the On-the-spot Reward, which provides immediate feedback on the accuracy of each tool invocation, and the Latent Reward, which assesses the contribution of each step toward overall task completion. By maximizing the cumulative reward of the On-the-spot and Latend Rewards at each step, LLMs are guided to follow efficient and accurate reasoning paths. Extensive experiments on StableToolBench and RestBench-TMDB demonstrate the superiority of CodeTool over existing approaches.