Conditionally Combining Robot Skills using Large Language Models
This work addresses the problem of enabling robots to generalize tasks with minimal data, though it is incremental by building on existing benchmarks and methods.
The paper introduces Language-World, an extension of the Meta-World benchmark for evaluating large language models in robotic environments, and Plan Conditioned Behavioral Cloning (PCBC), a method for fine-tuning high-level plans with demonstrations, achieving task generalization with as little as one demonstration.
This paper combines two contributions. First, we introduce an extension of the Meta-World benchmark, which we call "Language-World," which allows a large language model to operate in a simulated robotic environment using semi-structured natural language queries and scripted skills described using natural language. By using the same set of tasks as Meta-World, Language-World results can be easily compared to Meta-World results, allowing for a point of comparison between recent methods using Large Language Models (LLMs) and those using Deep Reinforcement Learning. Second, we introduce a method we call Plan Conditioned Behavioral Cloning (PCBC), that allows finetuning the behavior of high-level plans using end-to-end demonstrations. Using Language-World, we show that PCBC is able to achieve strong performance in a variety of few-shot regimes, often achieving task generalization with as little as a single demonstration. We have made Language-World available as open-source software at https://github.com/krzentner/language-world/.