Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
This work addresses the need for a versatile foundation model that can perform both understanding and generation tasks in vision-language AI, potentially reducing the need for multiple specialized models.
The paper tackles the problem of unifying multimodal understanding and generation by introducing Show-o, a single transformer that combines autoregressive and discrete diffusion modeling to handle mixed-modality inputs and outputs, achieving comparable or superior performance to specialized models across various vision-language benchmarks.
We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model. Code and models are released at https://github.com/showlab/Show-o.