Enhancing Q-Value Updates in Deep Q-Learning via Successor-State Prediction
This addresses a stability issue in deep reinforcement learning for researchers and practitioners, offering an incremental improvement over existing DQN methods.
The paper tackles the problem of high variance in Deep Q-Networks (DQNs) due to suboptimal next states from past policies, proposing the Successor-state Aggregation Deep Q-Network (SADQ) that integrates successor-state distributions for more stable Q-value updates. It shows that SADQ consistently outperforms DQN variants in stability and learning efficiency across benchmarks and real-world tasks.
Deep Q-Networks (DQNs) estimate future returns by learning from transitions sampled from a replay buffer. However, the target updates in DQN often rely on next states generated by actions from past, potentially suboptimal, policy. As a result, these states may not provide informative learning signals, causing high variance into the update process. This issue is exacerbated when the sampled transitions are poorly aligned with the agent's current policy. To address this limitation, we propose the Successor-state Aggregation Deep Q-Network (SADQ), which explicitly models environment dynamics using a stochastic transition model. SADQ integrates successor-state distributions into the Q-value estimation process, enabling more stable and policy-aligned value updates. Additionally, it explores a more efficient action selection strategy with the modeled transition structure. We provide theoretical guarantees that SADQ maintains unbiased value estimates while reducing training variance. Our extensive empirical results across standard RL benchmarks and real-world vector-based control tasks demonstrate that SADQ consistently outperforms DQN variants in both stability and learning efficiency.