Learning to Explore with Meta-Policy Gradient
This addresses a key bottleneck in reinforcement learning for researchers and practitioners by enabling more efficient training, though it is incremental as it builds on existing methods like DDPG.
The paper tackled the problem of inefficient exploration in off-policy reinforcement learning by developing a meta-policy gradient algorithm that learns adaptive exploration policies independent of the actor policy, resulting in global exploration that significantly speeds up learning and improves sample-efficiency on various tasks.
The performance of off-policy learning, including deep Q-learning and deep deterministic policy gradient (DDPG), critically depends on the choice of the exploration policy. Existing exploration methods are mostly based on adding noise to the on-going actor policy and can only explore \emph{local} regions close to what the actor policy dictates. In this work, we develop a simple meta-policy gradient algorithm that allows us to adaptively learn the exploration policy in DDPG. Our algorithm allows us to train flexible exploration behaviors that are independent of the actor policy, yielding a \emph{global exploration} that significantly speeds up the learning process. With an extensive study, we show that our method significantly improves the sample-efficiency of DDPG on a variety of reinforcement learning tasks.