Expert Q-learning: Deep Reinforcement Learning with Coarse State Values from Offline Expert Examples
This work addresses overestimation bias in reinforcement learning for applications like game AI, but it is incremental as it builds on existing Q-learning methods.
The authors tackled the problem of overestimation bias in deep reinforcement learning by proposing Expert Q-learning, which incorporates coarse state values from offline expert examples, resulting in more robust performance and higher scores compared to baseline Q-learning in non-deterministic settings like the board game Othello.
In this article, we propose a novel algorithm for deep reinforcement learning named Expert Q-learning. Expert Q-learning is inspired by Dueling Q-learning and aims at incorporating semi-supervised learning into reinforcement learning through splitting Q-values into state values and action advantages. We require that an offline expert assesses the value of a state in a coarse manner using three discrete values. An expert network is designed in addition to the Q-network, which updates each time following the regular offline minibatch update whenever the expert example buffer is not empty. Using the board game Othello, we compare our algorithm with the baseline Q-learning algorithm, which is a combination of Double Q-learning and Dueling Q-learning. Our results show that Expert Q-learning is indeed useful and more resistant to the overestimation bias. The baseline Q-learning algorithm exhibits unstable and suboptimal behavior in non-deterministic settings, whereas Expert Q-learning demonstrates more robust performance with higher scores, illustrating that our algorithm is indeed suitable to integrate state values from expert examples into Q-learning.