UNIC-Adapter: Unified Image-instruction Adapter with Multi-modal Transformer for Image Generation
This addresses the need for more precise and unified control in image generation for users in creative and design fields, representing a novel method rather than an incremental improvement.
The paper tackles the problem of achieving precise control over pixel-level layouts, object appearances, and global styles in text-to-image generation by proposing UNIC-Adapter, a unified framework that enables flexible and controllable generation across diverse conditions without multiple specialized models, with experimental results demonstrating its effectiveness across various tasks.
Recently, text-to-image generation models have achieved remarkable advancements, particularly with diffusion models facilitating high-quality image synthesis from textual descriptions. However, these models often struggle with achieving precise control over pixel-level layouts, object appearances, and global styles when using text prompts alone. To mitigate this issue, previous works introduce conditional images as auxiliary inputs for image generation, enhancing control but typically necessitating specialized models tailored to different types of reference inputs. In this paper, we explore a new approach to unify controllable generation within a single framework. Specifically, we propose the unified image-instruction adapter (UNIC-Adapter) built on the Multi-Modal-Diffusion Transformer architecture, to enable flexible and controllable generation across diverse conditions without the need for multiple specialized models. Our UNIC-Adapter effectively extracts multi-modal instruction information by incorporating both conditional images and task instructions, injecting this information into the image generation process through a cross-attention mechanism enhanced by Rotary Position Embedding. Experimental results across a variety of tasks, including pixel-level spatial control, subject-driven image generation, and style-image-based image synthesis, demonstrate the effectiveness of our UNIC-Adapter in unified controllable image generation.