Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
For researchers in embodied AI, gaming, and autonomous driving, this paper provides a structured overview of interactive world modeling, but it is a survey with no novel contributions.
This survey reviews recent progress in interactive world modeling, covering application scenarios, technical challenges (action-conditioned controllability, long-horizon interactions, real-time responsiveness), and benchmarks in open-world exploration, games, autonomous driving, and robotics. It identifies future directions but does not present new results or numbers.
With rapid development of large language models and diffusion-based content generation, world modeling has attracted increasing research attention, benefiting various downstream domains such as game engines, embodied AI, autonomous driving, etc. Through explicitly incorporating user actions into world state transition, recent literature empowers world modeling with interactivity in an action-conditioned video or 3D generation paradigm, further enhancing controllability over world evolutions and facilitating users to freely traverse, manipulate, navigate, and personalize the state evolution. In this paper, we aim to systematically review recent research trends, technical developments, evaluation benchmarks, and also propose future potential directions in interactive world modeling. Specifically, we first summarize recent efforts and trends in terms of application scenarios, world state evolution, and scene modality. Afterwards, we delve into three crucial technical challenges, including action-conditioned controllability, long-horizon interactions and memory, and action-following responsiveness for real-time interactivity. Furthermore, we also thoroughly compare existing benchmarks and metrics in four specific application fields: open-world exploration, game engine, autonomous driving, and robotics. Finally, we discuss several promising future directions in achieving next-generation interactive world modeling. The corresponding repository is publicly available at: https://github.com/liujiuming123/Awesome-Interactive-World-Model.