GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agents
This addresses the problem of scalable instruction-following for robotics and AI, offering a semi-supervised approach that reduces reliance on high-quality labeled data.
The paper tackles the challenge of training agents to follow multimodal instructions by introducing GROOT-2, a method that combines weak supervision with latent variable models, achieving robust performance across four diverse environments including video games and robotic manipulation.
Developing agents that can follow multimodal instructions remains a fundamental challenge in robotics and AI. Although large-scale pre-training on unlabeled datasets (no language instruction) has enabled agents to learn diverse behaviors, these agents often struggle with following instructions. While augmenting the dataset with instruction labels can mitigate this issue, acquiring such high-quality annotations at scale is impractical. To address this issue, we frame the problem as a semi-supervised learning task and introduce GROOT-2, a multimodal instructable agent trained using a novel approach that combines weak supervision with latent variable models. Our method consists of two key components: constrained self-imitating, which utilizes large amounts of unlabeled demonstrations to enable the policy to learn diverse behaviors, and human intention alignment, which uses a smaller set of labeled demonstrations to ensure the latent space reflects human intentions. GROOT-2's effectiveness is validated across four diverse environments, ranging from video games to robotic manipulation, demonstrating its robust multimodal instruction-following capabilities.