MMGen: Unified Multi-modal Image Generation and Understanding in One Go
This work addresses the need for seamless and controllable multi-modal image processing in applications requiring simultaneous generation and understanding, representing a novel method for a known bottleneck.
The paper tackles the problem of unifying multi-modal image generation and understanding tasks by introducing MMGen, a diffusion framework that integrates category-conditioned generation, visual understanding, and conditioned generation into a single model, achieving effectiveness across diverse tasks.
A unified diffusion framework for multi-modal generation and understanding has the transformative potential to achieve seamless and controllable image diffusion and other cross-modal tasks. In this paper, we introduce MMGen, a unified framework that integrates multiple generative tasks into a single diffusion model. This includes: (1) multi-modal category-conditioned generation, where multi-modal outputs are generated simultaneously through a single inference process, given category information; (2) multi-modal visual understanding, which accurately predicts depth, surface normals, and segmentation maps from RGB images; and (3) multi-modal conditioned generation, which produces corresponding RGB images based on specific modality conditions and other aligned modalities. Our approach develops a novel diffusion transformer that flexibly supports multi-modal output, along with a simple modality-decoupling strategy to unify various tasks. Extensive experiments and applications demonstrate the effectiveness and superiority of MMGen across diverse tasks and conditions, highlighting its potential for applications that require simultaneous generation and understanding.