PoM: Efficient Image and Video Generation with the Polynomial Mixer
This addresses efficiency bottlenecks for researchers and practitioners using diffusion models in computer vision, though it appears incremental as a replacement within existing architectures.
The authors tackled the quadratic computational and memory costs of Multi-Head Attention in diffusion models for image and video generation by proposing the Polynomial Mixer (PoM) as a drop-in replacement with linear complexity, achieving high-quality samples while using fewer computational resources.
Diffusion models based on Multi-Head Attention (MHA) have become ubiquitous to generate high quality images and videos. However, encoding an image or a video as a sequence of patches results in costly attention patterns, as the requirements both in terms of memory and compute grow quadratically. To alleviate this problem, we propose a drop-in replacement for MHA called the Polynomial Mixer (PoM) that has the benefit of encoding the entire sequence into an explicit state. PoM has a linear complexity with respect to the number of tokens. This explicit state also allows us to generate frames in a sequential fashion, minimizing memory and compute requirement, while still being able to train in parallel. We show the Polynomial Mixer is a universal sequence-to-sequence approximator, just like regular MHA. We adapt several Diffusion Transformers (DiT) for generating images and videos with PoM replacing MHA, and we obtain high quality samples while using less computational resources. The code is available at https://github.com/davidpicard/HoMM.