Align and Filter: Improving Performance in Asynchronous On-Policy RL
This addresses a bottleneck in scaling on-policy RL algorithms for broader applications, though it appears incremental as it builds on existing strategies to mitigate a known issue.
The paper tackled the problem of policy lag in asynchronous on-policy reinforcement learning, which arises from distributed training and high update frequencies, and proposed a method that improved robustness and performance in classic and modern RL tasks.
Distributed training and increasing the gradient update frequency are practical strategies to accelerate learning and improve performance, but both exacerbate a central challenge: \textit{policy lag}, which is the mismatch between the behavior policy generating data and the learning policy being updated. Policy lag can hinder the scaling of on-policy learning algorithms to larger problems. In this paper, we identify the sources of policy lag caused by distributed learning and high update frequency. We use the findings to propose \textit{total Variation-based Advantage aligned Constrained policy Optimization (\methodacronym)} as a practical approach to mitigate policy lag. We empirically validate our method and show that it offers better robustness to policy lag in classic RL tasks and a modern RL for LLM math reasoning task.