Continuous-Time Model-Based Reinforcement Learning
This work addresses the problem of accurately modeling continuous-time physical systems for reinforcement learning, which is important for researchers and practitioners working with real-world control tasks.
This paper proposes a continuous-time model-based reinforcement learning framework to address the limitations of discrete-time models in physical systems. The method uses Bayesian neural ordinary differential equations to infer state evolution differentials, demonstrating robustness against irregular and noisy data, sample efficiency, and the ability to solve challenging control problems for discrete-time methods.
Model-based reinforcement learning (MBRL) approaches rely on discrete-time state transition models whereas physical systems and the vast majority of control tasks operate in continuous-time. To avoid time-discretization approximation of the underlying process, we propose a continuous-time MBRL framework based on a novel actor-critic method. Our approach also infers the unknown state evolution differentials with Bayesian neural ordinary differential equations (ODE) to account for epistemic uncertainty. We implement and test our method on a new ODE-RL suite that explicitly solves continuous-time control systems. Our experiments illustrate that the model is robust against irregular and noisy data, is sample-efficient, and can solve control problems which pose challenges to discrete-time MBRL methods.