PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning
For researchers training LLM agents in long-horizon tasks, PATS offers a novel training-time support mechanism that adapts to policy evolution, reducing failures and improving efficiency.
PATS introduces a policy-aware training scaffold that dynamically adjusts rollout context for LLM agent reinforcement learning, improving task completion by up to 18.6% on ALFWorld and WebShop while using 32.1% fewer prompt tokens on QA benchmarks.
In long-horizon LLM agent reinforcement learning, weak policies often repeat similar failures, producing uninformative rollout trajectories and limiting effective policy optimization. Existing skill-centric methods improve exploration by optimizing, filtering, or internalizing reusable skills. However, they remain centered on the skills themselves rather than being designed as adaptive training-time support for the evolving policy. To address this, we propose a policy-centric training paradigm that reframes skills as a dynamic training scaffold. Our framework, Pats, converts rollout groups from the latest policy into evidence cards and uses task-specific evaluation to adjust the context used in subsequent rollouts. Concrete guidance helps weak policies to complete challenging tasks. As policy improves, redundant context is revised or removed to reduce reliance on explicit guidance while preserving useful rollout variation. The policy is optimized with environmental rewards using standard RLVR, and the training scaffold is discarded at deployment. On ALFWorld and WebShop, Pats improves over strong baselines by up to 18.6%. Across seven search-augmented QA benchmarks, it remains competitive while using 32.1% fewer prompt tokens than the baseline.