Grounded World Model for Semantically Generalizable Planning
For visuomotor control, this work enables semantic generalization to unseen visual and language instructions without requiring goal images, addressing a key limitation of existing methods.
The paper proposes a Grounded World Model (GWM) that aligns vision and language in a shared latent space for visuomotor MPC, enabling scoring of action proposals based on language instructions. On the WISER benchmark, GWM-MPC achieves 87% success on 288 unseen tasks, while traditional VLAs average 22% despite 90% on training.
In Model Predictive Control (MPC), world models predict the future outcomes of various action proposals, which are then scored to guide the selection of the optimal action. For visuomotor MPC, the score function is a distance metric between a predicted image and a goal image, measured in the latent space of a pretrained vision encoder like DINO and JEPA. However, it is challenging to obtain the goal image in advance of the task execution, particularly in new environments. Additionally, conveying the goal through an image offers limited interactivity compared with natural language. In this work, we propose to learn a Grounded World Model (GWM) in a vision-language-aligned latent space. As a result, each proposed action is scored based on how close its future outcome is to the task instruction, reflected by the similarity of embeddings. This approach transforms the visuomotor MPC to a VLA that surpasses VLM-based VLAs in semantic generalization. On the proposed WISER benchmark, GWM-MPC achieves a 87% success rate on the test set comprising 288 tasks that feature unseen visual signals and referring expressions, yet remain solvable with motions demonstrated during training. In contrast, traditional VLAs achieve an average success rate of 22%, even though they overfit the training set with a 90% success rate.