Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls
This work addresses optimal control in continuous-time systems for robotics or autonomous systems, offering a method that avoids discretizing dynamics and simplifies greedy action selection, though it appears incremental as an extension of deep Q-networks to continuous control.
The authors tackled the problem of continuous-time deterministic optimal control with Lipschitz continuous controls by proposing a Q-learning algorithm based on a novel Hamilton-Jacobi-Bellman equation, achieving convergence to the optimal Q-function under identified conditions and demonstrating performance on benchmark tasks and high-dimensional linear-quadratic problems.
In this paper, we propose Q-learning algorithms for continuous-time deterministic optimal control problems with Lipschitz continuous controls. Our method is based on a new class of Hamilton-Jacobi-Bellman (HJB) equations derived from applying the dynamic programming principle to continuous-time Q-functions. A novel semi-discrete version of the HJB equation is proposed to design a Q-learning algorithm that uses data collected in discrete time without discretizing or approximating the system dynamics. We identify the condition under which the Q-function estimated by this algorithm converges to the optimal Q-function. For practical implementation, we propose the Hamilton-Jacobi DQN, which extends the idea of deep Q-networks (DQN) to our continuous control setting. This approach does not require actor networks or numerical solutions to optimization problems for greedy actions since the HJB equation provides a simple characterization of optimal controls via ordinary differential equations. We empirically demonstrate the performance of our method through benchmark tasks and high-dimensional linear-quadratic problems.