Overcoming Overfitting in Reinforcement Learning via Gaussian Process Diffusion Policy
This addresses overfitting in RL for systems using deep neural networks, which is an incremental improvement with specific gains in robustness to distribution shifts.
The paper tackles the problem of overfitting in reinforcement learning due to distribution shifts by proposing the Gaussian Process Diffusion Policy (GPDP), which integrates diffusion models and Gaussian Process Regression to represent the policy, resulting in a 67.74% to 123.18% improvement in the objective function on the Walker2d benchmark under distribution shift conditions.
One of the key challenges that Reinforcement Learning (RL) faces is its limited capability to adapt to a change of data distribution caused by uncertainties. This challenge arises especially in RL systems using deep neural networks as decision makers or policies, which are prone to overfitting after prolonged training on fixed environments. To address this challenge, this paper proposes Gaussian Process Diffusion Policy (GPDP), a new algorithm that integrates diffusion models and Gaussian Process Regression (GPR) to represent the policy. GPR guides diffusion models to generate actions that maximize learned Q-function, resembling the policy improvement in RL. Furthermore, the kernel-based nature of GPR enhances the policy's exploration efficiency under distribution shifts at test time, increasing the chance of discovering new behaviors and mitigating overfitting. Simulation results on the Walker2d benchmark show that our approach outperforms state-of-the-art algorithms under distribution shift condition by achieving around 67.74% to 123.18% improvement in the RL's objective function while maintaining comparable performance under normal conditions.