Unified Discrete Diffusion for Simultaneous Vision-Language Generation
This work addresses the need for efficient multimodal AI systems capable of handling both modality translation and joint generation, though it appears incremental as it builds on existing discrete diffusion models.
The authors tackled the problem of multimodal generation by developing a unified discrete diffusion model that can perform text-to-image, image-to-text, and simultaneous vision-language generation tasks with a single model, achieving performance comparable to state-of-the-art solutions in various generation tasks.
The recently developed discrete diffusion models perform extraordinarily well in the text-to-image task, showing significant promise for handling the multi-modality signals. In this work, we harness these traits and present a unified multimodal generation model that can conduct both the "modality translation" and "multi-modality generation" tasks using a single model, performing text-based, image-based, and even vision-language simultaneous generation. Specifically, we unify the discrete diffusion process for multimodal signals by proposing a unified transition matrix. Moreover, we design a mutual attention module with fused embedding layer and a unified objective function to emphasise the inter-modal linkages, which are vital for multi-modality generation. Extensive experiments indicate that our proposed method can perform comparably to the state-of-the-art solutions in various generation tasks.