On Bellman equations for continuous-time policy evaluation I: discretization and approximation
This work addresses policy evaluation in continuous-time RL, offering improved error bounds for practitioners in control and simulation domains.
The authors tackled the problem of computing value functions from discretely-observed trajectories in continuous-time diffusion processes, developing new algorithms with high-order numerical accuracy and bounded approximation factors independent of horizon length.
We study the problem of computing the value function from a discretely-observed trajectory of a continuous-time diffusion process. We develop a new class of algorithms based on easily implementable numerical schemes that are compatible with discrete-time reinforcement learning (RL) with function approximation. We establish high-order numerical accuracy as well as the approximation error guarantees for the proposed approach. In contrast to discrete-time RL problems where the approximation factor depends on the effective horizon, we obtain a bounded approximation factor using the underlying elliptic structures, even if the effective horizon diverges to infinity.