Inter-environmental world modeling for continuous and compositional dynamics
This addresses the challenge of generalization across diverse environments for AI agents, offering a novel approach inspired by human capabilities, though it appears incremental in building on existing world model frameworks.
The paper tackles the problem of world models relying on discrete action representations by introducing WLA, an unsupervised framework that learns continuous latent actions to simulate across environments, achieving high controllability and predictive ability on synthetic and real-world datasets with minimal action labels.
Various world model frameworks are being developed today based on autoregressive frameworks that rely on discrete representations of actions and observations, and these frameworks are succeeding in constructing interactive generative models for the target environment of interest. Meanwhile, humans demonstrate remarkable generalization abilities to combine experiences in multiple environments to mentally simulate and learn to control agents in diverse environments. Inspired by this human capability, we introduce World modeling through Lie Action (WLA), an unsupervised framework that learns continuous latent action representations to simulate across environments. WLA learns a control interface with high controllability and predictive ability by simultaneously modeling the dynamics of multiple environments using Lie group theory and object-centric autoencoder. On synthetic benchmark and real-world datasets, we demonstrate that WLA can be trained using only video frames and, with minimal or no action labels, can quickly adapt to new environments with novel action sets.