Reinforcement Learning via Conservative Agent for Environments with Random Delays
This addresses a practical challenge for real-world RL applications where feedback delays are random, offering a solution that is incremental by extending constant-delay methods to random-delay scenarios.
The paper tackled the problem of reinforcement learning in environments with random delays, which violate the Markov assumption and are less explored than constant delays, by proposing a conservative agent that transforms random delays into constant delays, allowing existing methods to be applied without modification and achieving significant improvements in asymptotic performance and sample efficiency on continuous control tasks.
Real-world reinforcement learning applications are often hindered by delayed feedback from environments, which violates the Markov assumption and introduces significant challenges. Although numerous delay-compensating methods have been proposed for environments with constant delays, environments with random delays remain largely unexplored due to their inherent variability and unpredictability. In this study, we propose a simple yet robust agent for decision-making under random delays, termed the conservative agent, which reformulates the random-delay environment into its constant-delay equivalent. This transformation enables any state-of-the-art constant-delay method to be directly extended to the random-delay environments without modifying the algorithmic structure or sacrificing performance. We evaluate the conservative agent-based algorithm on continuous control tasks, and empirical results demonstrate that it significantly outperforms existing baseline algorithms in terms of asymptotic performance and sample efficiency.