Delay-Aware Model-Based Reinforcement Learning for Continuous Control
This addresses performance degradation in reinforcement learning for continuous control systems with delays, offering a novel framework but is incremental in building upon existing model-based approaches.
The paper tackles the problem of action delays degrading reinforcement learning performance in real-world systems by proposing a delay-aware model-based reinforcement learning framework, which proves more efficient in training and transferable across various delay durations compared to off-policy model-free methods.
Action delays degrade the performance of reinforcement learning in many real-world systems. This paper proposes a formal definition of delay-aware Markov Decision Process and proves it can be transformed into standard MDP with augmented states using the Markov reward process. We develop a delay-aware model-based reinforcement learning framework that can incorporate the multi-step delay into the learned system models without learning effort. Experiments with the Gym and MuJoCo platforms show that the proposed delay-aware model-based algorithm is more efficient in training and transferable between systems with various durations of delay compared with off-policy model-free reinforcement learning methods. Codes available at: https://github.com/baimingc/dambrl.