Policy Optimization for Continuous Reinforcement Learning
This work addresses the under-development of familiar RL methods like policy gradient and TRPO/PPO in continuous settings, which is incremental as it adapts existing tools to a new domain.
The authors tackled the problem of reinforcement learning in continuous time and space by developing a notion of occupation time for discounted objectives, which they applied to extend policy gradient and trust region methods from discrete to continuous RL, demonstrating effectiveness through numerical experiments.
We study reinforcement learning (RL) in the setting of continuous time and space, for an infinite horizon with a discounted objective and the underlying dynamics driven by a stochastic differential equation. Built upon recent advances in the continuous approach to RL, we develop a notion of occupation time (specifically for a discounted objective), and show how it can be effectively used to derive performance-difference and local-approximation formulas. We further extend these results to illustrate their applications in the PG (policy gradient) and TRPO/PPO (trust region policy optimization/ proximal policy optimization) methods, which have been familiar and powerful tools in the discrete RL setting but under-developed in continuous RL. Through numerical experiments, we demonstrate the effectiveness and advantages of our approach.