VUGEN: Visual Understanding priors for GENeration
This addresses the problem of misalignment between understanding and generation in VLMs for researchers and practitioners in computer vision, representing a strong specific gain rather than an incremental improvement.
The paper tackles the challenge of equipping Vision-Language Models with robust image generation capabilities by proposing VUGEN, a framework that leverages pretrained visual understanding priors, resulting in improved performance metrics such as DPG Bench from 71.17 to 74.32 and FID from 11.86 to 9.06 on COCO.
Recent advances in Vision-Language Models (VLMs) have enabled unified understanding across text and images, yet equipping these models with robust image generation capabilities remains challenging. Existing approaches often rely on reconstruction-oriented autoencoders or complex bridging mechanisms, leading to misalignment between understanding and generation representations, or architectural complexity. In this work, we propose VUGEN, a novel framework that explicitly leverages VLM's pretrained visual understanding priors for efficient and high-quality image generation. Our approach first transforms the high-dimensional latent space of the VLM's native vision encoder into a lower-dimensional, tractable distribution that maximally preserves visual information. The VLM is then trained to sample within this reduced latent space, ensuring alignment with its visual understanding capabilities. Finally, a dedicated pixel decoder maps these generated latents back to the image space. We find that a VAE-free pixel diffusion decoder to be on par or better than commonly used complex latent diffusion decoders that internally rely on VAE latents. Extensive experiments demonstrate that VUGEN achieves superior image generation performance, improving DPG Bench from 71.17 to 74.32 and FID from 11.86 to 9.06 on COCO, while fully preserving the VLM's original understanding capabilities.