Model-Augmented Actor-Critic: Backpropagating through Paths
This addresses a key bottleneck in reinforcement learning for applications requiring long-horizon planning, offering a more effective use of models beyond black-box simulation.
The paper tackles the problem of improving sample efficiency in model-based reinforcement learning by exploiting the differentiability of learned models to backpropagate through future timesteps, resulting in consistently higher sample efficiency than state-of-the-art model-based algorithms and matching the asymptotic performance of model-free methods.
Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator to augment the data for policy optimization or value function learning. In this paper, we show how to make more effective use of the model by exploiting its differentiability. We construct a policy optimization algorithm that uses the pathwise derivative of the learned model and policy across future timesteps. Instabilities of learning across many timesteps are prevented by using a terminal value function, learning the policy in an actor-critic fashion. Furthermore, we present a derivation on the monotonic improvement of our objective in terms of the gradient error in the model and value function. We show that our approach (i) is consistently more sample efficient than existing state-of-the-art model-based algorithms, (ii) matches the asymptotic performance of model-free algorithms, and (iii) scales to long horizons, a regime where typically past model-based approaches have struggled.