Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight
This addresses the challenge of enabling autonomous robots to follow natural language commands in real-world environments, though it appears incremental as it builds on existing simulation-to-real and hybrid learning approaches.
The researchers tackled the problem of mapping natural language instructions to continuous control for a physical quadcopter by developing a joint simulation and real-world learning framework that combines supervised and reinforcement learning. They demonstrated effective execution and exploration behavior on a physical quadcopter task.
We propose a joint simulation and real-world learning framework for mapping navigation instructions and raw first-person observations to continuous control. Our model estimates the need for environment exploration, predicts the likelihood of visiting environment positions during execution, and controls the agent to both explore and visit high-likelihood positions. We introduce Supervised Reinforcement Asynchronous Learning (SuReAL). Learning uses both simulation and real environments without requiring autonomous flight in the physical environment during training, and combines supervised learning for predicting positions to visit and reinforcement learning for continuous control. We evaluate our approach on a natural language instruction-following task with a physical quadcopter, and demonstrate effective execution and exploration behavior.