Factored Latent Action World Models
This work addresses the challenge of scaling controllable world models for complex environments with multiple entities, offering incremental improvements over existing latent action approaches.
The paper tackled the problem of learning controllable world models from action-free video in complex multi-entity environments, where monolithic models struggle. It introduced FLAM, a factored dynamics framework that decomposes scenes into independent factors, resulting in improved prediction accuracy and video generation quality compared to prior methods.
Learning latent actions from action-free video has emerged as a powerful paradigm for scaling up controllable world model learning. Latent actions provide a natural interface for users to iteratively generate and manipulate videos. However, most existing approaches rely on monolithic inverse and forward dynamics models that learn a single latent action to control the entire scene, and therefore struggle in complex environments where multiple entities act simultaneously. This paper introduces Factored Latent Action Model (FLAM), a factored dynamics framework that decomposes the scene into independent factors, each inferring its own latent action and predicting its own next-step factor value. This factorized structure enables more accurate modeling of complex multi-entity dynamics and improves video generation quality in action-free video settings compared to monolithic models. Based on experiments on both simulation and real-world multi-entity datasets, we find that FLAM outperforms prior work in prediction accuracy and representation quality, and facilitates downstream policy learning, demonstrating the benefits of factorized latent action models.