Proximal Policy Optimization with Mixed Distributed Training
This is an incremental improvement to PPO for reinforcement learning practitioners.
The paper tackles instability and slowness in deep reinforcement learning by introducing Mixed Distributed Proximal Policy Optimization (MDPPO), which accelerates and stabilizes training, particularly in sparse-reward environments.
Instability and slowness are two main problems in deep reinforcement learning. Even if proximal policy optimization (PPO) is the state of the art, it still suffers from these two problems. We introduce an improved algorithm based on proximal policy optimization, mixed distributed proximal policy optimization (MDPPO), and show that it can accelerate and stabilize the training process. In our algorithm, multiple different policies train simultaneously and each of them controls several identical agents that interact with environments. Actions are sampled by each policy separately as usual, but the trajectories for the training process are collected from all agents, instead of only one policy. We find that if we choose some auxiliary trajectories elaborately to train policies, the algorithm will be more stable and quicker to converge especially in the environments with sparse rewards.