Token Pruning for Caching Better: 9 Times Acceleration on Stable Diffusion for Free
This work addresses the computational bottleneck in text-to-image generation for broader adoption, representing an incremental improvement by combining feature caching with token pruning.
The paper tackles the high computational cost and slow generation speed of Stable Diffusion by introducing a dynamics-aware token pruning (DaTo) approach that selectively prunes low-dynamic tokens to extend feature dynamics across timesteps, achieving up to 9x speedup on ImageNet with a 0.33 FID reduction and 7x acceleration on COCO-30k with a 2.17 FID reduction.
Stable Diffusion has achieved remarkable success in the field of text-to-image generation, with its powerful generative capabilities and diverse generation results making a lasting impact. However, its iterative denoising introduces high computational costs and slows generation speed, limiting broader adoption. The community has made numerous efforts to reduce this computational burden, with methods like feature caching attracting attention due to their effectiveness and simplicity. Nonetheless, simply reusing features computed at previous timesteps causes the features across adjacent timesteps to become similar, reducing the dynamics of features over time and ultimately compromising the quality of generated images. In this paper, we introduce a dynamics-aware token pruning (DaTo) approach that addresses the limitations of feature caching. DaTo selectively prunes tokens with lower dynamics, allowing only high-dynamic tokens to participate in self-attention layers, thereby extending feature dynamics across timesteps. DaTo combines feature caching with token pruning in a training-free manner, achieving both temporal and token-wise information reuse. Applied to Stable Diffusion on the ImageNet, our approach delivered a 9$\times$ speedup while reducing FID by 0.33, indicating enhanced image quality. On the COCO-30k, we observed a 7$\times$ acceleration coupled with a notable FID reduction of 2.17.