Approximation to Deep Q-Network by Stochastic Delay Differential Equations
This provides theoretical insights into DQN's stability mechanisms for reinforcement learning researchers, though it is incremental as it builds on existing DQN analysis.
The paper tackled the limited theoretical analysis of Deep Q-Network (DQN) by constructing a stochastic differential delay equation (SDDE) to approximate it, proving that the Wasserstein-1 distance between them converges to zero as step size decreases, which helps interpret DQN's experience replay and target network techniques in continuous terms.
Despite the significant breakthroughs that the Deep Q-Network (DQN) has brought to reinforcement learning, its theoretical analysis remains limited. In this paper, we construct a stochastic differential delay equation (SDDE) based on the DQN algorithm and estimate the Wasserstein-1 distance between them. We provide an upper bound for the distance and prove that the distance between the two converges to zero as the step size approaches zero. This result allows us to understand DQN's two key techniques, the experience replay and the target network, from the perspective of continuous systems. Specifically, the delay term in the equation, corresponding to the target network, contributes to the stability of the system. Our approach leverages a refined Lindeberg principle and an operator comparison to establish these results.