Guiding Token-Sparse Diffusion Models
This addresses the efficiency bottleneck for users of diffusion models in image synthesis, offering significant computational savings while maintaining quality, though it is an incremental improvement over existing sparse training methods.
The paper tackles the problem of high computational cost in diffusion models during inference by proposing Sparse Guidance (SG), which uses token-level sparsity to improve response to guidance, achieving a 1.58 FID on ImageNet-256 with 25% fewer FLOPs and up to 58% FLOP savings.
Diffusion models deliver high quality in image synthesis but remain expensive during training and inference. Recent works have leveraged the inherent redundancy in visual content to make training more affordable by training only on a subset of visual information. While these methods were successful in providing cheaper and more effective training, sparsely trained diffusion models struggle in inference. This is due to their lacking response to Classifier-free Guidance (CFG) leading to underwhelming performance during inference. To overcome this, we propose Sparse Guidance (SG). Instead of using conditional dropout as a signal to guide diffusion models, SG uses token-level sparsity. As a result, SG preserves the high-variance of the conditional prediction better, achieving good quality and high variance outputs. Leveraging token-level sparsity at inference, SG improves fidelity at lower compute, achieving 1.58 FID on the commonly used ImageNet-256 benchmark with 25% fewer FLOPs, and yields up to 58% FLOP savings at matched baseline quality. To demonstrate the effectiveness of Sparse Guidance, we train a 2.5B text-to-image diffusion model using training time sparsity and leverage SG during inference. SG achieves improvements in composition and human preference score while increasing throughput at the same time.