Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time
This addresses the challenge of applying reinforcement learning to continuous-time control problems, which is incremental as it adapts discrete-time Q-learning methods to a continuous-time framework.
The paper tackles the problem of extending Q-learning to continuous-time optimal control by introducing Hamilton-Jacobi-Bellman (HJB) equations for Q-functions, showing that the Q-function is the unique viscosity solution of the HJB equation and developing a Q-learning method with a DQN-like algorithm, demonstrated on 1-, 10-, and 20-dimensional dynamical systems.
In this paper, we introduce Hamilton-Jacobi-Bellman (HJB) equations for Q-functions in continuous time optimal control problems with Lipschitz continuous controls. The standard Q-function used in reinforcement learning is shown to be the unique viscosity solution of the HJB equation. A necessary and sufficient condition for optimality is provided using the viscosity solution framework. By using the HJB equation, we develop a Q-learning method for continuous-time dynamical systems. A DQN-like algorithm is also proposed for high-dimensional state and control spaces. The performance of the proposed Q-learning algorithm is demonstrated using 1-, 10- and 20-dimensional dynamical systems.