Cross Learning in Deep Q-Networks
This addresses a known bottleneck in reinforcement learning for AI agents, but it is incremental as it builds on double Q-learning.
The paper tackles the overestimation problem in deep Q-networks by proposing a cross Q-learning algorithm that uses parallel models to reduce bias and variance, leading to improved performance and more stable training in benchmark environments.
In this work, we propose a novel cross Q-learning algorithm, aim at alleviating the well-known overestimation problem in value-based reinforcement learning methods, particularly in the deep Q-networks where the overestimation is exaggerated by function approximation errors. Our algorithm builds on double Q-learning, by maintaining a set of parallel models and estimate the Q-value based on a randomly selected network, which leads to reduced overestimation bias as well as the variance. We provide empirical evidence on the advantages of our method by evaluating on some benchmark environment, the experimental results demonstrate significant improvement of performance in reducing the overestimation bias and stabilizing the training, further leading to better derived policies.