Some approaches used to overcome overestimation in Deep Reinforcement Learning algorithms
This is an incremental analysis of noise effects in RL algorithms, relevant for researchers working on algorithm stability and performance.
The paper discusses overestimation issues in deep reinforcement learning algorithms, examining DQN, double DQN, DDPG, TD3, and hill climbing, and explores noise parameter settings for TD3 in PyBullet environments like HopperBulletEnv and Walker2DBulletEnv.
Some phenomena related to statistical noise which have been investigated by various authors under the framework of deep reinforcement learning (RL) algorithms are discussed. The following algorithms are examined: the deep Q-network (DQN), double DQN, deep deterministic policy gradient (DDPG), twin-delayed DDPG (TD3), and hill climbing algorithm. First, we consider overestimation, which is a harmful property resulting from noise. Then we deal with noise used for exploration, this is the useful noise. We discuss setting the noise parameter in the TD3 for typical PyBullet environments associated with articulate bodies such as HopperBulletEnv and Walker2DBulletEnv. In the appendix, in relation to the hill climbing algorithm, another example related to noise is considered - an example of adaptive noise.