LATTE: LAnguage Trajectory TransformEr
This work addresses the challenge of generalizing language-based robotic control across different hardware platforms, which is a significant problem for robotics researchers and practitioners, though it builds incrementally on previous work by extending trajectory parametrization and using scene images.
The paper tackles the problem of translating free-form language instructions into robotic trajectories by proposing a flexible framework that uses pre-trained language models and transformer networks to encode intent and output trajectories without task-specific priors. The result is a system that successfully modifies trajectory shape and speed across diverse robotic scenarios, including manipulation, aerial, and legged robots, as demonstrated in simulated and real-life experiments.
Natural language is one of the most intuitive ways to express human intent. However, translating instructions and commands towards robotic motion generation and deployment in the real world is far from being an easy task. The challenge of combining a robot's inherent low-level geometric and kinodynamic constraints with a human's high-level semantic instructions traditionally is solved using task-specific solutions with little generalizability between hardware platforms, often with the use of static sets of target actions and commands. This work instead proposes a flexible language-based framework that allows a user to modify generic robotic trajectories. Our method leverages pre-trained language models (BERT and CLIP) to encode the user's intent and target objects directly from a free-form text input and scene images, fuses geometrical features generated by a transformer encoder network, and finally outputs trajectories using a transformer decoder, without the need of priors related to the task or robot information. We significantly extend our own previous work presented in Bucker et al. by expanding the trajectory parametrization space to 3D and velocity as opposed to just XY movements. In addition, we now train the model to use actual images of the objects in the scene for context (as opposed to textual descriptions), and we evaluate the system in a diverse set of scenarios beyond manipulation, such as aerial and legged robots. Our simulated and real-life experiments demonstrate that our transformer model can successfully follow human intent, modifying the shape and speed of trajectories within multiple environments. Codebase available at: https://github.com/arthurfenderbucker/LaTTe-Language-Trajectory-TransformEr.git