Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agents
This addresses efficiency and performance issues in tool-using LLM agents for real-world applications, representing an incremental improvement.
The paper tackled the problem of excessive and low-quality tool calls in LLM-based agents during long trajectories, which increases latency and degrades performance, by proposing entropy reduction as a supervisory signal with two reward strategies; experiments showed a 72.07% reduction in tool calls and a 22.27% performance improvement.
Tool-using agents based on Large Language Models (LLMs) excel in tasks such as mathematical reasoning and multi-hop question answering. However, in long trajectories, agents often trigger excessive and low-quality tool calls, increasing latency and degrading inference performance, making managing tool-use behavior challenging. In this work, we conduct entropy-based pilot experiments and observe a strong positive correlation between entropy reduction and high-quality tool calls. Building on this finding, we propose using entropy reduction as a supervisory signal and design two reward strategies to address the differing needs of optimizing tool-use behavior. Sparse outcome rewards provide coarse, trajectory-level guidance to improve efficiency, while dense process rewards offer fine-grained supervision to enhance performance. Experiments across diverse domains show that both reward designs improve tool-use behavior: the former reduces tool calls by 72.07% compared to the average of baselines, while the latter improves performance by 22.27%. These results position entropy reduction as a key mechanism for enhancing tool-use behavior, enabling agents to be more adaptive in real-world applications.