Probing Implicit Bias in Semi-gradient Q-learning: Visualizing the Effective Loss Landscapes via the Fokker--Planck Equation
This work addresses the problem of understanding implicit bias in semi-gradient Q-learning for researchers in reinforcement learning, though it is incremental as it builds on existing methods with a novel visualization approach.
The paper tackled the challenge of studying the dynamics and implicit bias of semi-gradient Q-learning, which lacks an explicit loss function, by introducing the Fokker–Planck equation to construct and visualize effective loss landscapes in parameter spaces. The result revealed that global minima in the loss landscape can transform into saddle points in the effective landscape, and this phenomenon persists in high-dimensional and neural network settings.
Semi-gradient Q-learning is applied in many fields, but due to the absence of an explicit loss function, studying its dynamics and implicit bias in the parameter space is challenging. This paper introduces the Fokker--Planck equation and employs partial data obtained through sampling to construct and visualize the effective loss landscape within a two-dimensional parameter space. This visualization reveals how the global minima in the loss landscape can transform into saddle points in the effective loss landscape, as well as the implicit bias of the semi-gradient method. Additionally, we demonstrate that saddle points, originating from the global minima in loss landscape, still exist in the effective loss landscape under high-dimensional parameter spaces and neural network settings. This paper develop a novel approach for probing implicit bias in semi-gradient Q-learning.