Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents
This work addresses a fundamental problem in learning dynamics for LLM agents in long-horizon tasks, offering a novel method to improve efficiency and stability, though it is incremental in the context of policy gradient techniques.
The paper tackles the challenge of sparse rewards in long-horizon tasks for LLM-based agents by proposing Entropy-Modulated Policy Gradients (EMPG), which re-calibrates learning signals based on uncertainty and task outcomes, achieving substantial performance gains and outperforming strong baselines on tasks like WebShop, ALFWorld, and Deep Search.
In long-horizon tasks, recent agents based on Large Language Models (LLMs) face a significant challenge that sparse, outcome-based rewards make it difficult to assign credit to intermediate steps. Previous methods mainly focus on creating dense reward signals to guide learning, either through traditional reinforcement learning techniques like inverse reinforcement learning or by using Process Reward Models for step-by-step feedback. In this paper, we identify a fundamental problem in the learning dynamics of LLMs: the magnitude of policy gradients is inherently coupled with the entropy, which leads to inefficient small updates for confident correct actions and potentially destabilizes large updates for uncertain ones. To resolve this, we propose Entropy-Modulated Policy Gradients (EMPG), a framework that re-calibrates the learning signal based on step-wise uncertainty and the final task outcome. EMPG amplifies updates for confident correct actions, penalizes confident errors, and attenuates updates from uncertain steps to stabilize exploration. We further introduce a bonus term for future clarity that encourages agents to find more predictable solution paths. Through comprehensive experiments on three challenging agent tasks, WebShop, ALFWorld, and Deep Search, we demonstrate that EMPG achieves substantial performance gains and significantly outperforms strong policy gradient baselines. Project page is at https://empgseed-seed.github.io/