Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control
This addresses a long-standing challenge in continuous-time RL for domains requiring precise control, though it appears incremental as it builds on existing score-based and Q-function concepts.
The authors tackled the problem of reinforcement learning in continuous-time control by introducing Continuous Q-Score Matching (CQSM), a method that links diffusion policy scores to Q-functions without time discretization, achieving effective performance in simulated environments.
Reinforcement learning (RL) has achieved significant success across a wide range of domains, however, most existing methods are formulated in discrete time. In this work, we introduce a novel RL method for continuous-time control, where stochastic differential equations govern state-action dynamics. Departing from traditional value function-based approaches, our key contribution is the characterization of continuous-time Q-functions via a martingale condition and the linking of diffusion policy scores to the action gradient of a learned continuous Q-function by the dynamic programming principle. This insight motivates Continuous Q-Score Matching (CQSM), a score-based policy improvement algorithm. Notably, our method addresses a long-standing challenge in continuous-time RL: preserving the action-evaluation capability of Q-functions without relying on time discretization. We further provide theoretical closed-form solutions for linear-quadratic (LQ) control problems within our framework. Numerical results in simulated environments demonstrate the effectiveness of our proposed method and compare it to popular baselines.