Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning
This addresses a bottleneck in LLM-based agent frameworks for AI researchers, though it appears incremental as it builds on existing reinforcement learning and agent methods.
The paper tackles the challenge of simultaneously handling tool calling and reasoning processes in LLM-based agents by proposing a hierarchical framework called Agent-as-tool, which separates these processes to reduce reasoning burden; it achieved 63.2% exact match and 75.2% cover exact match on Bamboogle, exceeding Search-R1 by 4.8% and 3.2% respectively.
Large Language Models (LLMs) have emerged as one of the most significant technological advancements in artificial intelligence in recent years. Their ability to understand, generate, and reason with natural language has transformed how we interact with AI systems. With the development of LLM-based agents and reinforcement-learning-based reasoning models, the study of applying reinforcement learning in agent frameworks has become a new research focus. However, all previous studies face the challenge of deciding the tool calling process and the reasoning process simultaneously, and the chain of reasoning was solely relied on the unprocessed raw result with redundant information and symbols unrelated to the task from the tool, which impose a heavy burden on the model's capability to reason. Therefore, in our research, we proposed a hierarchical framework Agent-as-tool that detach the tool calling process and the reasoning process, which enables the model to focus on the verbally reasoning process while the tool calling process is handled by another agent. Our work had achieved comparable results with only a slight reinforcement fine-tuning on 180 samples, and had achieved exceptionally well performance in Bamboogle with 63.2% of exact match and 75.2% in cover exact match, exceeding Search-R1 by 4.8% in exact match and 3.2% in cover exact match.