Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models
This work addresses the efficiency and accessibility problem for researchers and practitioners using text-to-image models, offering a more cost-effective and environmentally friendly alternative to existing state-of-the-art methods.
The paper tackles the high computational cost of large-scale text-to-image diffusion models by introducing Wuerstchen, an architecture that uses a compact semantic image representation to guide diffusion, reducing training time by over 87% (from 200,000 to 24,602 A100-GPU hours) and doubling inference speed while maintaining competitive image quality.
We introduce Würstchen, a novel architecture for text-to-image synthesis that combines competitive performance with unprecedented cost-effectiveness for large-scale text-to-image diffusion models. A key contribution of our work is to develop a latent diffusion technique in which we learn a detailed but extremely compact semantic image representation used to guide the diffusion process. This highly compressed representation of an image provides much more detailed guidance compared to latent representations of language and this significantly reduces the computational requirements to achieve state-of-the-art results. Our approach also improves the quality of text-conditioned image generation based on our user preference study. The training requirements of our approach consists of 24,602 A100-GPU hours - compared to Stable Diffusion 2.1's 200,000 GPU hours. Our approach also requires less training data to achieve these results. Furthermore, our compact latent representations allows us to perform inference over twice as fast, slashing the usual costs and carbon footprint of a state-of-the-art (SOTA) diffusion model significantly, without compromising the end performance. In a broader comparison against SOTA models our approach is substantially more efficient and compares favorably in terms of image quality. We believe that this work motivates more emphasis on the prioritization of both performance and computational accessibility.