Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization
For reinforcement learning practitioners dealing with real-world delayed feedback, DUPO provides a robust method that outperforms prior approaches, especially under long and random delays.
DUPO addresses performance degradation in reinforcement learning due to delayed feedback by modeling the discrepancy between delayed and true states with a diffusion model, achieving consistent improvements over existing methods on continuous robotic control tasks with stochastic delays.
Reinforcement learning in real world environments often suffers from severe performance degradation due to delayed feedback. Existing approaches typically mitigate performance degradation caused by observation delays by constructing augmented states or predicting the true states. However, these methods often overlook the inherent discrepancy between delayed state and true states induced by stochastic MDP. We theoretically prove the existence of such a discrepancy and show that it leads to the degradation of the optimal policy. To address this challenge, we propose Diffusion Guided Uncertainty Aware Delayed Policy Optimization (DUPO). Our method explicitly models the relationship between delayed state message and the current state using a diffusion model, and leverages the resulting discrepancy estimates to weight delayed policies. Extensive experiments on continuous robotic control tasks with multiple stochastic delays demonstrate that DUPO consistently outperforms existing methods and remains effective even under long and random delay scenarios.