Mobile Video Diffusion
This work addresses the problem of deploying realistic video generation on mobile devices, representing an incremental improvement by optimizing an existing model for efficiency.
The paper tackles the high computational demands of video diffusion models by introducing MobileVD, a mobile-optimized model that reduces efficiency by 523x (from 1817.2 to 4.34 TFLOPs) with a slight quality drop (FVD from 149 to 171), generating video clips in 1.7 seconds on a mobile device.
Video diffusion models have achieved impressive realism and controllability but are limited by high computational demands, restricting their use on mobile devices. This paper introduces the first mobile-optimized video diffusion model. Starting from a spatio-temporal UNet from Stable Video Diffusion (SVD), we reduce memory and computational cost by reducing the frame resolution, incorporating multi-scale temporal representations, and introducing two novel pruning schema to reduce the number of channels and temporal blocks. Furthermore, we employ adversarial finetuning to reduce the denoising to a single step. Our model, coined as MobileVD, is 523x more efficient (1817.2 vs. 4.34 TFLOPs) with a slight quality drop (FVD 149 vs. 171), generating latents for a 14x512x256 px clip in 1.7 seconds on a Xiaomi-14 Pro. Our results are available at https://qualcomm-ai-research.github.io/mobile-video-diffusion/