Multi-Architecture Multi-Expert Diffusion Models
This work addresses the problem of balancing efficiency and quality in diffusion models for image generation, offering an incremental design choice that can be applied to other scenarios like large multi-expert models.
The paper tackles performance degradation in efficient diffusion models by introducing Multi-Architecture Multi-Expert diffusion models (MEME), which assign distinct architectures to different time-step intervals, resulting in a 3.3x speedup and improved FID scores by 0.62 on FFHQ and 0.37 on CelebA.
In this paper, we address the performance degradation of efficient diffusion models by introducing Multi-architecturE Multi-Expert diffusion models (MEME). We identify the need for tailored operations at different time-steps in diffusion processes and leverage this insight to create compact yet high-performing models. MEME assigns distinct architectures to different time-step intervals, balancing convolution and self-attention operations based on observed frequency characteristics. We also introduce a soft interval assignment strategy for comprehensive training. Empirically, MEME operates 3.3 times faster than baselines while improving image generation quality (FID scores) by 0.62 (FFHQ) and 0.37 (CelebA). Though we validate the effectiveness of assigning more optimal architecture per time-step, where efficient models outperform the larger models, we argue that MEME opens a new design choice for diffusion models that can be easily applied in other scenarios, such as large multi-expert models.