Target-Aligned Reinforcement Learning
This addresses a fundamental bottleneck in reinforcement learning training for researchers and practitioners, though it is incremental as it builds on existing target network methods.
The paper tackles the stability-recency tradeoff in reinforcement learning caused by target networks, proposing Target-Aligned Reinforcement Learning (TARL) to focus updates on well-aligned targets, which accelerates convergence and shows consistent improvements in benchmark environments.
Many reinforcement learning algorithms rely on target networks - lagged copies of the online network - to stabilize training. While effective, this mechanism introduces a fundamental stability-recency tradeoff: slower target updates improve stability but reduce the recency of learning signals, hindering convergence speed. We propose Target-Aligned Reinforcement Learning (TARL), a framework that emphasizes transitions for which the target and online network estimates are highly aligned. By focusing updates on well-aligned targets, TARL mitigates the adverse effects of stale target estimates while retaining the stabilizing benefits of target networks. We provide a theoretical analysis demonstrating that target alignment correction accelerates convergence, and empirically demonstrate consistent improvements over standard reinforcement learning algorithms across various benchmark environments.