Towards Natural Language-Driven Assembly Using Foundation Models
This addresses the problem of enabling natural language-driven robotic control for industrial assembly tasks, offering a domain-specific incremental improvement over existing generalist policies.
The paper tackles the challenge of using generalist vision-language-action policies for high-precision robotic assembly tasks like insertion, which require accuracy and handling of contact and friction. It presents a method using LLMs to transfer control to a finite set of specifically trained skills, achieving improved precision in assembly operations.
Large Language Models (LLMs) and strong vision models have enabled rapid research and development in the field of Vision-Language-Action models that enable robotic control. The main objective of these methods is to develop a generalist policy that can control robots with various embodiments. However, in industrial robotic applications such as automated assembly and disassembly, some tasks, such as insertion, demand greater accuracy and involve intricate factors like contact engagement, friction handling, and refined motor skills. Implementing these skills using a generalist policy is challenging because these policies might integrate further sensory data, including force or torque measurements, for enhanced precision. In our method, we present a global control policy based on LLMs that can transfer the control policy to a finite set of skills that are specifically trained to perform high-precision tasks through dynamic context switching. The integration of LLMs into this framework underscores their significance in not only interpreting and processing language inputs but also in enriching the control mechanisms for diverse and intricate robotic operations.