Deep Q-Learning with Gradient Target Tracking
This addresses a specific bottleneck in reinforcement learning for practitioners by providing a more automated and stable training alternative, though it is incremental as it builds on existing DQN frameworks.
The paper tackles the challenge of manually tuning the hard update period in deep Q-networks (DQN) by introducing gradient-based target tracking methods, which replace periodic hard updates with continuous gradient descent updates, eliminating the need for tuning and showing empirical advantages over standard DQN baselines.
This paper introduces Q-learning with gradient target tracking, a novel reinforcement learning framework that provides a learned continuous target update mechanism as an alternative to the conventional hard update paradigm. In the standard deep Q-network (DQN), the target network is a copy of the online network's weights, held fixed for a number of iterations before being periodically replaced via a hard update. While this stabilizes training by providing consistent targets, it introduces a new challenge: the hard update period must be carefully tuned to achieve optimal performance. To address this issue, we propose two gradient-based target update methods: DQN with asymmetric gradient target tracking (AGT2-DQN) and DQN with symmetric gradient target tracking (SGT2-DQN). These methods replace the conventional hard target updates with continuous and structured updates using gradient descent, which effectively eliminates the need for manual tuning. We provide a theoretical analysis proving the convergence of these methods in tabular settings. Additionally, empirical evaluations demonstrate their advantages over standard DQN baselines, which suggest that gradient-based target updates can serve as an effective alternative to conventional target update mechanisms in Q-learning.