SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spaces
This work addresses the computational bottleneck in video generation for researchers and practitioners by enabling more efficient long-term video synthesis, though it is incremental as it adapts existing SSM methods to a new domain.
The paper tackles the challenge of generating long-term videos efficiently by replacing attention layers with structured state-space models (SSMs) as temporal feature extractors, resulting in SSM-based models requiring less memory to achieve the same Fréchet Video Distance (FVD) as attention-based models for sequences up to 256 frames and often delivering better performance with comparable GPU usage.
Given the remarkable achievements in image generation through diffusion models, the research community has shown increasing interest in extending these models to video generation. Recent diffusion models for video generation have predominantly utilized attention layers to extract temporal features. However, attention layers are limited by their computational costs, which increase quadratically with the sequence length. This limitation presents significant challenges when generating longer video sequences using diffusion models. To overcome this challenge, we propose leveraging state-space models (SSMs) as temporal feature extractors. SSMs (e.g., Mamba) have recently gained attention as promising alternatives due to their linear-time memory consumption relative to sequence length. In line with previous research suggesting that using bidirectional SSMs is effective for understanding spatial features in image generation, we found that bidirectionality is also beneficial for capturing temporal features in video data, rather than relying on traditional unidirectional SSMs. We conducted comprehensive evaluations on multiple long-term video datasets, such as MineRL Navigate, across various model sizes. For sequences up to 256 frames, SSM-based models require less memory to achieve the same FVD as attention-based models. Moreover, SSM-based models often deliver better performance with comparable GPU memory usage. Our codes are available at https://github.com/shim0114/SSM-Meets-Video-Diffusion-Models.