Tube-NeRF: Efficient Imitation Learning of Visuomotor Policies from MPC using Tube-Guided Data Augmentation and NeRFs
This work addresses the problem of training efficient and robust vision-based control policies for robotics, particularly multirotors, by improving imitation learning methods, though it is incremental as it builds on existing MPC and NeRF techniques.
The paper tackles the inefficiency and robustness issues in imitation learning of visuomotor policies from MPC by proposing Tube-NeRF, a data augmentation method that uses NeRFs and robust MPC to generate synthetic images and actions, resulting in an 80-fold increase in demonstration efficiency, 50% reduction in training time, and successful real-world deployment with low tracking errors and 1.5 ms inference time.
Imitation learning (IL) can train computationally-efficient sensorimotor policies from a resource-intensive Model Predictive Controller (MPC), but it often requires many samples, leading to long training times or limited robustness. To address these issues, we combine IL with a variant of robust MPC that accounts for process and sensing uncertainties, and we design a data augmentation (DA) strategy that enables efficient learning of vision-based policies. The proposed DA method, named Tube-NeRF, leverages Neural Radiance Fields (NeRFs) to generate novel synthetic images, and uses properties of the robust MPC (the tube) to select relevant views and to efficiently compute the corresponding actions. We tailor our approach to the task of localization and trajectory tracking on a multirotor, by learning a visuomotor policy that generates control actions using images from the onboard camera as only source of horizontal position. Numerical evaluations show 80-fold increase in demonstration efficiency and a 50% reduction in training time over current IL methods. Additionally, our policies successfully transfer to a real multirotor, achieving low tracking errors despite large disturbances, with an onboard inference time of only 1.5 ms. Video: https://youtu.be/_W5z33ZK1m4