World Model Implanting for Test-time Adaptation of Embodied Agents
This addresses the problem of cross-domain adaptability for embodied AI agents, offering a scalable solution for real-world deployment, though it appears incremental as it builds on existing LLM and world model techniques.
The paper tackles the challenge of enabling embodied agents to adapt to novel domains without extensive data or retraining by introducing a world model implanting framework (WorMI) that combines LLMs with domain-specific world models, achieving superior zero-shot and few-shot performance on VirtualHome and ALFWorld benchmarks.
In embodied AI, a persistent challenge is enabling agents to robustly adapt to novel domains without requiring extensive data collection or retraining. To address this, we present a world model implanting framework (WorMI) that combines the reasoning capabilities of large language models (LLMs) with independently learned, domain-specific world models through test-time composition. By allowing seamless implantation and removal of the world models, the embodied agent's policy achieves and maintains cross-domain adaptability. In the WorMI framework, we employ a prototype-based world model retrieval approach, utilizing efficient trajectory-based abstract representation matching, to incorporate relevant models into test-time composition. We also develop a world-wise compound attention method that not only integrates the knowledge from the retrieved world models but also aligns their intermediate representations with the reasoning model's representation within the agent's policy. This framework design effectively fuses domain-specific knowledge from multiple world models, ensuring robust adaptation to unseen domains. We evaluate our WorMI on the VirtualHome and ALFWorld benchmarks, demonstrating superior zero-shot and few-shot performance compared to several LLM-based approaches across a range of unseen domains. These results highlight the frameworks potential for scalable, real-world deployment in embodied agent scenarios where adaptability and data efficiency are essential.