Modified Actor-Critics
This provides a new generic tool for deep reinforcement learning, potentially improving efficiency for practitioners, though it appears incremental as it builds on existing methods like PPO and MPI.
The paper tackles the sample inefficiency of on-policy deep reinforcement learning algorithms by proposing a framework that combines soft greediness with Modified Policy Iteration, enabling off-policy learning. The result is a more sample-efficient algorithm that is competitive with state-of-the-art off-policy methods like SAC.
Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are naturally on-policy. In this paper, we propose to combine (any kind of) soft greediness with Modified Policy Iteration (MPI). The proposed abstract framework applies repeatedly: (i) a partial policy evaluation step that allows off-policy learning and (ii) any softened greedy step. Our contribution can be seen as a new generic tool for the deep reinforcement learning toolbox. As a proof of concept, we instantiate this framework with the PPO greediness. Comparison to the original PPO shows that our algorithm is much more sample efficient. We also show that it is competitive with the state-of-art off-policy algorithm Soft Actor Critic (SAC).