Exploring More When It Needs in Deep Reinforcement Learning
This addresses the challenge of exploration in deep reinforcement learning for continuous control tasks, but it is incremental as it builds on existing methods like DDPG and SAC.
The paper tackles the problem of inefficient exploration in deep reinforcement learning by proposing an exploration mechanism called Add Noise to Noise (AN2N), which increases exploration when the agent has historically performed poorly, and shows performance and convergence speed improvements on continuous control tasks like halfCheetah, Hopper, and Swimmer.
We propose a exploration mechanism of policy in Deep Reinforcement Learning, which is exploring more when agent needs, called Add Noise to Noise (AN2N). The core idea is: when the Deep Reinforcement Learning agent is in a state of poor performance in history, it needs to explore more. So we use cumulative rewards to evaluate which past states the agents have not performed well, and use cosine distance to measure whether the current state needs to be explored more. This method shows that the exploration mechanism of the agent's policy is conducive to efficient exploration. We combining the proposed exploration mechanism AN2N with Deep Deterministic Policy Gradient (DDPG), Soft Actor-Critic (SAC) algorithms, and apply it to the field of continuous control tasks, such as halfCheetah, Hopper, and Swimmer, achieving considerable improvement in performance and convergence speed.