Regularizing Trajectory Optimization with Denoising Autoencoders
This incremental improvement addresses sample efficiency issues for researchers and practitioners in reinforcement learning.
The paper tackles the problem of model inaccuracies in trajectory optimization for model-based reinforcement learning by proposing a denoising autoencoder as a regularization method, resulting in improved planning and rapid initial learning in motor control tasks.
Trajectory optimization using a learned model of the environment is one of the core elements of model-based reinforcement learning. This procedure often suffers from exploiting inaccuracies of the learned model. We propose to regularize trajectory optimization by means of a denoising autoencoder that is trained on the same trajectories as the model of the environment. We show that the proposed regularization leads to improved planning with both gradient-based and gradient-free optimizers. We also demonstrate that using regularized trajectory optimization leads to rapid initial learning in a set of popular motor control tasks, which suggests that the proposed approach can be a useful tool for improving sample efficiency.