CARL: Focusing Agentic Reinforcement Learning on Critical Actions
This work addresses inefficiencies in long-horizon agentic reasoning for AI agents, representing an incremental improvement in reinforcement learning methods.
The paper tackles the problem of suboptimal policy optimization in multi-step agentic reinforcement learning by proposing CARL, an algorithm that focuses training on critical actions using entropy as a proxy, resulting in stronger performance and higher efficiency across diverse settings.
Agents capable of accomplishing complex tasks through multiple interactions with the environment have emerged as a popular research direction. However, in such multi-step settings, the conventional group-level policy optimization algorithm becomes suboptimal because of its underlying assumption that each action holds equal contribution, which deviates significantly from reality. Our analysis reveals that only a small fraction of actions are critical in determining the final outcome. Building on this insight, we propose CARL, a critical-action-focused reinforcement learning algorithm tailored for long-horizon agentic reasoning. CARL leverages entropy as a heuristic proxy for action criticality and achieves focused training by assigning rewards to high-criticality actions while excluding low-criticality actions from model updates, avoiding noisy credit assignment and redundant computation. Extensive experiments demonstrate that CARL achieves both stronger performance and higher efficiency across diverse evaluation settings. The source code will be publicly available.