Data-Efficient Learning of Feedback Policies from Image Pixels using Deep Dynamical Models
This addresses the problem of enabling autonomous systems to learn control policies efficiently from visual inputs, representing an incremental advance in model-based RL for robotics.
The paper tackles the challenge of data-efficient reinforcement learning from high-dimensional pixel observations, specifically the pixels-to-torques problem, by introducing a model-based algorithm that learns a closed-loop policy directly from pixels, achieving quick learning and scalability to high-dimensional states.
Data-efficient reinforcement learning (RL) in continuous state-action spaces using very high-dimensional observations remains a key challenge in developing fully autonomous systems. We consider a particularly important instance of this challenge, the pixels-to-torques problem, where an RL agent learns a closed-loop control policy ("torques") from pixel information only. We introduce a data-efficient, model-based reinforcement learning algorithm that learns such a closed-loop policy directly from pixel information. The key ingredient is a deep dynamical model for learning a low-dimensional feature embedding of images jointly with a predictive model in this low-dimensional feature space. Joint learning is crucial for long-term predictions, which lie at the core of the adaptive nonlinear model predictive control strategy that we use for closed-loop control. Compared to state-of-the-art RL methods for continuous states and actions, our approach learns quickly, scales to high-dimensional state spaces, is lightweight and an important step toward fully autonomous end-to-end learning from pixels to torques.