Instruction-driven history-aware policies for robotic manipulations
This addresses the problem of enabling robots to perform complex manipulation tasks in human environments using simple instructions, representing a strong incremental advance in robotics.
The paper tackles robotic manipulation by proposing a transformer-based approach that integrates natural language instructions, multi-view scene observations, and full history tracking to improve precision and generalization. It scales to 74 diverse tasks on the RLBench benchmark, outperforming state-of-the-art methods and showing excellent generalization to unseen variations.
In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions. Yet, robotic manipulation is extremely challenging as it requires fine-grained motor control, long-term memory as well as generalization to previously unseen tasks and environments. To address these challenges, we propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full history of observations and actions. Such an approach enables learning dependencies between history and instructions and improves manipulation precision using multiple views. We evaluate our method on the challenging RLBench benchmark and on a real-world robot. Notably, our approach scales to 74 diverse RLBench tasks and outperforms the state of the art. We also address instruction-conditioned tasks and demonstrate excellent generalization to previously unseen variations.