Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model
This work addresses the challenge of unifying generative and discriminative modeling in computer vision, offering a novel approach for researchers and practitioners, though it is incremental in combining existing architectures.
The paper tackles the problem of connecting diffusion models and Vision Transformers by integrating ViT into DDPM, resulting in Generative ViT and Hybrid ViT models that achieve state-of-the-art performance in both generative and discriminative tasks.
Diffusion Denoising Probability Models (DDPM) and Vision Transformer (ViT) have demonstrated significant progress in generative tasks and discriminative tasks, respectively, and thus far these models have largely been developed in their own domains. In this paper, we establish a direct connection between DDPM and ViT by integrating the ViT architecture into DDPM, and introduce a new generative model called Generative ViT (GenViT). The modeling flexibility of ViT enables us to further extend GenViT to hybrid discriminative-generative modeling, and introduce a Hybrid ViT (HybViT). Our work is among the first to explore a single ViT for image generation and classification jointly. We conduct a series of experiments to analyze the performance of proposed models and demonstrate their superiority over prior state-of-the-arts in both generative and discriminative tasks. Our code and pre-trained models can be found in https://github.com/sndnyang/Diffusion_ViT .