Reinforcement Learning with Random Delays
It addresses delays in reinforcement learning for applications like remote control, but appears incremental as it builds on existing methods.
The paper tackles the problem of action and observation delays in reinforcement learning, particularly in remote control scenarios, by proposing Delay-Correcting Actor-Critic (DCAC), which shows significantly better performance on a delay-augmented MuJoCo benchmark.
Action and observation delays commonly occur in many Reinforcement Learning applications, such as remote control scenarios. We study the anatomy of randomly delayed environments, and show that partially resampling trajectory fragments in hindsight allows for off-policy multi-step value estimation. We apply this principle to derive Delay-Correcting Actor-Critic (DCAC), an algorithm based on Soft Actor-Critic with significantly better performance in environments with delays. This is shown theoretically and also demonstrated practically on a delay-augmented version of the MuJoCo continuous control benchmark.