UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and Perception
This work addresses the problem of computational inefficiency and fragmented designs in multimodal AI for researchers and practitioners, representing an incremental improvement by unifying tasks within a single model.
The paper tackles the challenge of fragmented and inefficient architectures for multimodal tasks like image generation, understanding, manipulation, and perception by proposing UniAlignment, a unified framework using a single diffusion transformer with dual-stream training for semantic alignment. It outperforms existing baselines across multiple tasks and benchmarks, as demonstrated in extensive experiments.
The remarkable success of diffusion models in text-to-image generation has sparked growing interest in expanding their capabilities to a variety of multi-modal tasks, including image understanding, manipulation, and perception. These tasks require advanced semantic comprehension across both visual and textual modalities, especially in scenarios involving complex semantic instructions. However, existing approaches often rely heavily on vision-language models (VLMs) or modular designs for semantic guidance, leading to fragmented architectures and computational inefficiency. To address these challenges, we propose UniAlignment, a unified multimodal generation framework within a single diffusion transformer. UniAlignment introduces a dual-stream diffusion training strategy that incorporates both intrinsic-modal semantic alignment and cross-modal semantic alignment, thereby enhancing the model's cross-modal consistency and instruction-following robustness. Additionally, we present SemGen-Bench, a new benchmark specifically designed to evaluate multimodal semantic consistency under complex textual instructions. Extensive experiments across multiple tasks and benchmarks demonstrate that UniAlignment outperforms existing baselines, underscoring the significant potential of diffusion models in unified multimodal generation.