Adapting Offline Reinforcement Learning with Online Delays
This addresses the sim-to-real and interaction gaps for deploying RL agents in real-world systems where latency is a challenge, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the problem of deploying offline reinforcement learning agents in environments with online delays, which break the Markov assumption and degrade performance, by introducing DT-CORL, a framework that uses a transformer-based belief predictor to produce delay-robust actions without seeing delayed data during training, and it consistently outperforms baselines on D4RL benchmarks with various delay settings.
Offline-to-online deployment of reinforcement-learning (RL) agents must bridge two gaps: (1) the sim-to-real gap, where real systems add latency and other imperfections not present in simulation, and (2) the interaction gap, where policies trained purely offline face out-of-distribution states during online execution because gathering new interaction data is costly or risky. Agents therefore have to generalize from static, delay-free datasets to dynamic, delay-prone environments. Standard offline RL learns from delay-free logs yet must act under delays that break the Markov assumption and hurt performance. We introduce DT-CORL (Delay-Transformer belief policy Constrained Offline RL), an offline-RL framework built to cope with delayed dynamics at deployment. DT-CORL (i) produces delay-robust actions with a transformer-based belief predictor even though it never sees delayed observations during training, and (ii) is markedly more sample-efficient than naïve history-augmentation baselines. Experiments on D4RL benchmarks with several delay settings show that DT-CORL consistently outperforms both history-augmentation and vanilla belief-based methods, narrowing the sim-to-real latency gap while preserving data efficiency.