Q-learning for Optimal Control of Continuous-time Systems
This work addresses optimal control for continuous-time systems, offering a model-free approach that could benefit robotics or automation, but it appears incremental as it adapts existing Q-learning frameworks to a specific domain.
The authors tackled the model-free optimal control problem for nonlinear continuous-time systems by proposing two Q-learning methods, PIQL and VIQL, which converge to the optimal Q-function and learn optimal control policies from real system data, as verified through computer simulation.
In this paper, two Q-learning (QL) methods are proposed and their convergence theories are established for addressing the model-free optimal control problem of general nonlinear continuous-time systems. By introducing the Q-function for continuous-time systems, policy iteration based QL (PIQL) and value iteration based QL (VIQL) algorithms are proposed for learning the optimal control policy from real system data rather than using mathematical system model. It is proved that both PIQL and VIQL methods generate a nonincreasing Q-function sequence, which converges to the optimal Q-function. For implementation of the QL algorithms, the method of weighted residuals is applied to derived the parameters update rule. The developed PIQL and VIQL algorithms are essentially off-policy reinforcement learning approachs, where the system data can be collected arbitrary and thus the exploration ability is increased. With the data collected from the real system, the QL methods learn the optimal control policy offline, and then the convergent control policy will be employed to real system. The effectiveness of the developed QL algorithms are verified through computer simulation.