Deep Spatial Autoencoders for Visuomotor Learning
This addresses the challenge of requiring detailed state representations for robotic motion skills, though it is incremental as it builds on existing reinforcement learning and autoencoder methods.
The paper tackles the problem of automating state-space construction for robotic reinforcement learning by learning a state representation directly from camera images, resulting in a controller that enables a PR2 robot to perform tasks like pushing a block, picking up a bag of rice, and hanging rope on a hook with closed-loop control.
Reinforcement learning provides a powerful and flexible framework for automated acquisition of robotic motion skills. However, applying reinforcement learning requires a sufficiently detailed representation of the state, including the configuration of task-relevant objects. We present an approach that automates state-space construction by learning a state representation directly from camera images. Our method uses a deep spatial autoencoder to acquire a set of feature points that describe the environment for the current task, such as the positions of objects, and then learns a motion skill with these feature points using an efficient reinforcement learning method based on local linear models. The resulting controller reacts continuously to the learned feature points, allowing the robot to dynamically manipulate objects in the world with closed-loop control. We demonstrate our method with a PR2 robot on tasks that include pushing a free-standing toy block, picking up a bag of rice using a spatula, and hanging a loop of rope on a hook at various positions. In each task, our method automatically learns to track task-relevant objects and manipulate their configuration with the robot's arm.