Natural Gradient Deep Q-learning
This work addresses hyperparameter tuning and stability issues in deep reinforcement learning for researchers and practitioners, though it is incremental as it builds on existing DQN methods.
The paper tackled the problem of training deep Q-learning agents by introducing a natural-gradient variant (NGDQN), which significantly outperformed DQN without target networks and matched DQN with target networks in classic control domains, stabilizing training and reducing hyperparameter sensitivity.
We present a novel algorithm to train a deep Q-learning agent using natural-gradient techniques. We compare the original deep Q-network (DQN) algorithm to its natural-gradient counterpart, which we refer to as NGDQN, on a collection of classic control domains. Without employing target networks, NGDQN significantly outperforms DQN without target networks, and performs no worse than DQN with target networks, suggesting that NGDQN stabilizes training and can help reduce the need for additional hyperparameter tuning. We also find that NGDQN is less sensitive to hyperparameter optimization relative to DQN. Together these results suggest that natural-gradient techniques can improve value-function optimization in deep reinforcement learning.