Accelerated Continuous-Time Approximate Dynamic Programming via Data-Assisted Hybrid Control
This work addresses control problems for continuous-time dynamic plants with affine input structures, offering an incremental improvement by accelerating convergence and relaxing excitation requirements.
The paper tackles the online solution of approximate optimal control for continuous-time systems by introducing a closed-loop architecture with dynamic momentum in actor-critic structures, achieving accelerated convergence and superior transient performance compared to traditional gradient-descent methods, while eliminating the need for persistence of excitation conditions using past data.
We introduce a new closed-loop architecture for the online solution of approximate optimal control problems in the context of continuous-time systems. Specifically, we introduce the first algorithm that incorporates dynamic momentum in actor-critic structures to control continuous-time dynamic plants with an affine structure in the input. By incorporating dynamic momentum in our algorithm, we are able to accelerate the convergence properties of the closed-loop system, achieving superior transient performance compared to traditional gradient-descent based techniques. In addition, by leveraging the existence of past recorded data with sufficiently rich information properties, we dispense with the persistence of excitation condition traditionally imposed on the regressors of the critic and the actor. Given that our continuous-time momentum-based dynamics also incorporate periodic discrete-time resets that emulate restarting techniques used in the machine learning literature, we leverage tools from hybrid dynamical systems theory to establish asymptotic stability properties for the closed-loop system. We illustrate our results with a numerical example.