GP-ILQG: Data-driven Robust Optimal Control for Uncertain Nonlinear Dynamical Systems
This work addresses the problem of simulation-to-reality transfer for controlling complex systems, offering a hybrid approach that is incremental in nature.
The paper tackles the Reality Gap in model-based reinforcement learning for uncertain nonlinear dynamical systems by combining Gaussian Process system identification with robust optimal control, resulting in an algorithm that quickly corrects model errors and efficiently transfers knowledge to new tasks.
As we aim to control complex systems, use of a simulator in model-based reinforcement learning is becoming more common. However, it has been challenging to overcome the Reality Gap, which comes from nonlinear model bias and susceptibility to disturbance. To address these problems, we propose a novel algorithm that combines data-driven system identification approach (Gaussian Process) with a Differential-Dynamic-Programming-based robust optimal control method (Iterative Linear Quadratic Control). Our algorithm uses the simulator's model as the mean function for a Gaussian Process and learns only the difference between the simulator's prediction and actual observations, making it a natural hybrid of simulation and real-world observation. We show that our approach quickly corrects incorrect models, comes up with robust optimal controllers, and transfers its acquired model knowledge to new tasks efficiently.