Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Models
This addresses the challenge of systematic compositionality in visual world models for machine learning researchers, though it is incremental as it focuses on benchmarking rather than a new method.
The paper introduces the Systematic Visual Imagination Benchmark (SVIB) to evaluate models on systematic visual imagination, specifically generating one-step image-to-image transformations under latent world dynamics, and provides a comprehensive evaluation of baseline models to assess current capabilities.
Systematic compositionality, or the ability to adapt to novel situations by creating a mental model of the world using reusable pieces of knowledge, remains a significant challenge in machine learning. While there has been considerable progress in the language domain, efforts towards systematic visual imagination, or envisioning the dynamical implications of a visual observation, are in their infancy. We introduce the Systematic Visual Imagination Benchmark (SVIB), the first benchmark designed to address this problem head-on. SVIB offers a novel framework for a minimal world modeling problem, where models are evaluated based on their ability to generate one-step image-to-image transformations under a latent world dynamics. The framework provides benefits such as the possibility to jointly optimize for systematic perception and imagination, a range of difficulty levels, and the ability to control the fraction of possible factor combinations used during training. We provide a comprehensive evaluation of various baseline models on SVIB, offering insight into the current state-of-the-art in systematic visual imagination. We hope that this benchmark will help advance visual systematic compositionality.