RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancy
This work addresses the computational bottleneck in video generation for AI researchers and practitioners, offering a plug-and-play acceleration method that is incremental but broadly applicable to state-of-the-art models.
The paper tackles the high computational cost of video generation with diffusion models, where 3D attention accounts for over 80% of resources, by introducing RainFusion, a training-free sparse attention method that achieves over 2× speedup in attention computation while maintaining video quality with minimal impact on VBench scores (-0.2%).
Video generation using diffusion models is highly computationally intensive, with 3D attention in Diffusion Transformer (DiT) models accounting for over 80\% of the total computational resources. In this work, we introduce {\bf RainFusion}, a novel training-free sparse attention method that exploits inherent sparsity nature in visual data to accelerate attention computation while preserving video quality. Specifically, we identify three unique sparse patterns in video generation attention calculations--Spatial Pattern, Temporal Pattern and Textural Pattern. The sparse pattern for each attention head is determined online with negligible overhead (\textasciitilde\,0.2\%) with our proposed {\bf ARM} (Adaptive Recognition Module) during inference. Our proposed {\bf RainFusion} is a plug-and-play method, that can be seamlessly integrated into state-of-the-art 3D-attention video generation models without additional training or calibration. We evaluate our method on leading open-sourced models including HunyuanVideo, OpenSoraPlan-1.2 and CogVideoX-5B, demonstrating its broad applicability and effectiveness. Experimental results show that RainFusion achieves over {\bf 2\(\times\)} speedup in attention computation while maintaining video quality, with only a minimal impact on VBench scores (-0.2\%).