TBAC-UniImage: Unified Understanding and Generation by Ladder-Side Diffusion Tuning
This addresses limitations in unified multimodal models for researchers and practitioners, though it appears incremental as it builds on existing diffusion and MLLM methods.
The paper tackles the problem of shallow connections and high computational cost in diffusion-based unified models for multimodal understanding and generation by introducing TBAC-UniImage, which uses representations from multiple layers of a Multimodal Large Language Model as conditions for a diffusion model, achieving a deeper unification.
This paper introduces TBAC-UniImage, a novel unified model for multimodal understanding and generation. We achieve this by deeply integrating a pre-trained Diffusion Model, acting as a generative ladder, with a Multimodal Large Language Model (MLLM). Previous diffusion-based unified models face two primary limitations. One approach uses only the MLLM's final hidden state as the generative condition. This creates a shallow connection, as the generator is isolated from the rich, hierarchical representations within the MLLM's intermediate layers. The other approach, pretraining a unified generative architecture from scratch, is computationally expensive and prohibitive for many researchers. To overcome these issues, our work explores a new paradigm. Instead of relying on a single output, we use representations from multiple, diverse layers of the MLLM as generative conditions for the diffusion model. This method treats the pre-trained generator as a ladder, receiving guidance from various depths of the MLLM's understanding process. Consequently, TBAC-UniImage achieves a much deeper and more fine-grained unification of understanding and generation.