Grounding Multimodal Large Language Models in Actions
This work addresses the challenge of adapting MLLMs for embodied AI applications, representing an incremental improvement in grounding methods.
The paper tackles the problem of grounding multimodal large language models (MLLMs) into different embodiments and action spaces to leverage their world knowledge, showing that learned tokenization for continuous actions and semantic alignment for discrete actions yield the best performance on downstream tasks across over 114 embodied tasks.
Multimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains, including Embodied AI. In this work, we study how to best ground a MLLM into different embodiments and their associated action spaces, with the goal of leveraging the multimodal world knowledge of the MLLM. We first generalize a number of methods through a unified architecture and the lens of action space adaptors. For continuous actions, we show that a learned tokenization allows for sufficient modeling precision, yielding the best performance on downstream tasks. For discrete actions, we demonstrate that semantically aligning these actions with the native output token space of the MLLM leads to the strongest performance. We arrive at these lessons via a thorough study of seven action space adapters on five different environments, encompassing over 114 embodied tasks.