State-Action Inpainting Diffuser for Continuous Control with Delay
This addresses the fundamental challenge of delay in RL for control systems, proposing a novel hybrid methodology that could advance the field.
The paper tackles the problem of signal delay in continuous control and reinforcement learning by introducing the State-Action Inpainting Diffuser (SAID), which integrates dynamics learning with policy optimization to generate consistent plans, achieving state-of-the-art performance on delayed benchmarks.
Signal delay poses a fundamental challenge in continuous control and reinforcement learning (RL) by introducing a temporal gap between interaction and perception. Current solutions have largely evolved along two distinct paradigms: model-free approaches which utilize state augmentation to preserve Markovian properties, and model-based methods which focus on inferring latent beliefs via dynamics modeling. In this paper, we bridge these perspectives by introducing State-Action Inpainting Diffuser (SAID), a framework that integrates the inductive bias of dynamics learning with the direct decision-making capability of policy optimization. By formulating the problem as a joint sequence inpainting task, SAID implicitly captures environmental dynamics while directly generating consistent plans, effectively operating at the intersection of model-based and model-free paradigms. Crucially, this generative formulation allows SAID to be seamlessly applied to both online and offline RL. Extensive experiments on delayed continuous control benchmarks demonstrate that SAID achieves state-of-the-art and robust performance. Our study suggests a new methodology to advance the field of RL with delay.