VisualCloze: A Universal Image Generation Framework via Visual In-Context Learning
This addresses the need for more flexible and efficient image generation models in AI, though it is incremental as it builds on existing diffusion and in-context learning methods.
The authors tackled the problem of building a universal image generation model that can handle multiple tasks without task-specific designs, proposing VisualCloze, which uses visual in-context learning and a graph-structured dataset to achieve generalization across seen and unseen tasks, with results showing improved efficiency and task unification.
Recent progress in diffusion models significantly advances various image generation tasks. However, the current mainstream approach remains focused on building task-specific models, which have limited efficiency when supporting a wide range of different needs. While universal models attempt to address this limitation, they face critical challenges, including generalizable task instruction, appropriate task distributions, and unified architectural design. To tackle these challenges, we propose VisualCloze, a universal image generation framework, which supports a wide range of in-domain tasks, generalization to unseen ones, unseen unification of multiple tasks, and reverse generation. Unlike existing methods that rely on language-based task instruction, leading to task ambiguity and weak generalization, we integrate visual in-context learning, allowing models to identify tasks from visual demonstrations. Meanwhile, the inherent sparsity of visual task distributions hampers the learning of transferable knowledge across tasks. To this end, we introduce Graph200K, a graph-structured dataset that establishes various interrelated tasks, enhancing task density and transferable knowledge. Furthermore, we uncover that our unified image generation formulation shared a consistent objective with image infilling, enabling us to leverage the strong generative priors of pre-trained infilling models without modifying the architectures.