Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learning
This addresses the challenge of enabling household robots to handle under-specified instructions more effectively, though it is an incremental advance in applying existing techniques to a new domain.
The paper tackles the problem of embodied agents interpreting ambiguous human instructions in household environments by introducing the Ask-to-Act task, where agents ask clarification questions to resolve ambiguity, and proposes a method that fine-tunes multimodal LLMs with reinforcement learning using LLM-generated rewards, resulting in a 10.4-16.5% performance improvement over baselines.
Embodied agents operating in household environments must interpret ambiguous and under-specified human instructions. A capable household robot should recognize ambiguity and ask relevant clarification questions to infer the user intent accurately, leading to more effective task execution. To study this problem, we introduce the Ask-to-Act task, where an embodied agent is tasked with a single or multi-object rearrangement task using an under-specified instruction in a home environment. The agent must strategically ask minimal, yet relevant, clarification questions to resolve ambiguity while navigating under partial observability. To address this challenge, we propose a novel approach that fine-tunes multi-modal large language models (MLLMs) as vision-language-action (VLA) policies using online reinforcement learning (RL) with LLM-generated rewards. Our method eliminates the need for large-scale human demonstrations or manually engineered rewards for training such agents. We benchmark against strong zero-shot baselines including GPT-4o as well as supervised fine-tuned MLLMs on our task. Our results show that our RL-finetuned MLLM outperforms all baselines by a significant margin (10.4-16.5%), generalizing well to novel scenes and tasks. To the best of our knowledge, this is the first demonstration of adapting MLLMs as VLA agents that can act and ask for help using LLM-generated rewards with online RL.