Multimodal Representation Alignment for Image Generation: Text-Image Interleaved Control Is Easier Than You Think
This addresses the problem of merging concepts from multiple images in generation for AI researchers and practitioners, representing an incremental advance by building on existing text-to-image models.
The paper tackles the lack of comprehensive frameworks for arbitrary text-image interleaved control in image generation by proposing Dream Engine, which uses large multimodal models to align image and text representations as conditions for diffusion models, achieving a 0.69 overall score on the GenEval benchmark and matching state-of-the-art models like SD3.5 and FLUX.
The field of advanced text-to-image generation is witnessing the emergence of unified frameworks that integrate powerful text encoders, such as CLIP and T5, with Diffusion Transformer backbones. Although there have been efforts to control output images with additional conditions, like canny and depth map, a comprehensive framework for arbitrary text-image interleaved control is still lacking. This gap is especially evident when attempting to merge concepts or visual elements from multiple images in the generation process. To mitigate the gap, we conducted preliminary experiments showing that large multimodal models (LMMs) offer an effective shared representation space, where image and text can be well-aligned to serve as a condition for external diffusion models. Based on this discovery, we propose Dream Engine, an efficient and unified framework designed for arbitrary text-image interleaved control in image generation models. Building on powerful text-to-image models like SD3.5, we replace the original text-only encoders by incorporating versatile multimodal information encoders such as QwenVL. Our approach utilizes a two-stage training paradigm, consisting of joint text-image alignment and multimodal interleaved instruction tuning. Our experiments demonstrate that this training method is effective, achieving a 0.69 overall score on the GenEval benchmark, and matching the performance of state-of-the-art text-to-image models like SD3.5 and FLUX.