Yuan-TecSwin: A text conditioned Diffusion model with Swin-transformer blocks
This work addresses a bottleneck in text-to-image generation for AI researchers and practitioners, though it is incremental as it builds on existing diffusion and transformer architectures.
The authors tackled the problem of limited long-range semantic understanding in diffusion models for image synthesis by replacing CNN blocks with Swin-transformer blocks, resulting in a state-of-the-art FID score of 1.37 on ImageNet and a 10% improvement in inference performance.
Diffusion models have shown remarkable capacity in image synthesis based on their U-shaped architecture and convolutional neural networks (CNN) as basic blocks. The locality of the convolution operation in CNN may limit the model's ability to understand long-range semantic information. To address this issue, we propose Yuan-TecSwin, a text-conditioned diffusion model with Swin-transformer in this work. The Swin-transformer blocks take the place of CNN blocks in the encoder and decoder, to improve the non-local modeling ability in feature extraction and image restoration. The text-image alignment is improved with a well-chosen text encoder, effective utilization of text embedding, and careful design in the incorporation of text condition. Using an adapted time step to search in different diffusion stages, inference performance is further improved by 10%. Yuan-TecSwin achieves the state-of-the-art FID score of 1.37 on ImageNet generation benchmark, without any additional models at different denoising stages. In a side-by-side comparison, we find it difficult for human interviewees to tell the model-generated images from the human-painted ones.