Non-local Policy Optimization via Diversity-regularized Collaborative Exploration
This work addresses exploration inefficiencies in reinforcement learning for robotics and control tasks, offering an incremental improvement through collaborative agent teamwork.
The paper tackled the problem of limited exploration and local minima in reinforcement learning by introducing a non-local policy optimization framework using a team of heterogeneous agents with diversity regularization, achieving substantial improvement over baselines in MuJoCo locomotion tasks.
Conventional Reinforcement Learning (RL) algorithms usually have one single agent learning to solve the task independently. As a result, the agent can only explore a limited part of the state-action space while the learned behavior is highly correlated to the agent's previous experience, making the training prone to a local minimum. In this work, we empower RL with the capability of teamwork and propose a novel non-local policy optimization framework called Diversity-regularized Collaborative Exploration (DiCE). DiCE utilizes a group of heterogeneous agents to explore the environment simultaneously and share the collected experiences. A regularization mechanism is further designed to maintain the diversity of the team and modulate the exploration. We implement the framework in both on-policy and off-policy settings and the experimental results show that DiCE can achieve substantial improvement over the baselines in the MuJoCo locomotion tasks.