Regret Analysis of Certainty Equivalence Policies in Continuous-Time Linear-Quadratic Systems
This provides theoretical guarantees for a common policy in continuous-time control, addressing exploration-exploitation in stochastic systems, though it is incremental as it builds on existing certainty equivalence methods.
The paper tackles the problem of controlling continuous-time linear-quadratic systems with unknown dynamics using reinforcement learning, showing that a randomized certainty equivalent policy achieves square-root of time regret bounds and linear scaling with parameters.
This work theoretically studies a ubiquitous reinforcement learning policy for controlling the canonical model of continuous-time stochastic linear-quadratic systems. We show that randomized certainty equivalent policy addresses the exploration-exploitation dilemma in linear control systems that evolve according to unknown stochastic differential equations and their operating cost is quadratic. More precisely, we establish square-root of time regret bounds, indicating that randomized certainty equivalent policy learns optimal control actions fast from a single state trajectory. Further, linear scaling of the regret with the number of parameters is shown. The presented analysis introduces novel and useful technical approaches, and sheds light on fundamental challenges of continuous-time reinforcement learning.