Exploring Vision Transformers as Diffusion Learners
This work addresses the modeling choices and training pipelines for diffusion-based generative models, which is an incremental improvement for the computer vision and AI research community.
The paper tackles the problem of improving diffusion models for vision generative tasks by exploring vision Transformers as backbones, achieving performance on par with traditional U-Net-based methods and competitive results on datasets like CIFAR-10 and CelebA, including training a single diffusion model beyond 64x64 resolution for text-to-image tasks.
Score-based diffusion models have captured widespread attention and funded fast progress of recent vision generative tasks. In this paper, we focus on diffusion model backbone which has been much neglected before. We systematically explore vision Transformers as diffusion learners for various generative tasks. With our improvements the performance of vanilla ViT-based backbone (IU-ViT) is boosted to be on par with traditional U-Net-based methods. We further provide a hypothesis on the implication of disentangling the generative backbone as an encoder-decoder structure and show proof-of-concept experiments verifying the effectiveness of a stronger encoder for generative tasks with ASymmetriC ENcoder Decoder (ASCEND). Our improvements achieve competitive results on CIFAR-10, CelebA, LSUN, CUB Bird and large-resolution text-to-image tasks. To the best of our knowledge, we are the first to successfully train a single diffusion model on text-to-image task beyond 64x64 resolution. We hope this will motivate people to rethink the modeling choices and the training pipelines for diffusion-based generative models.