LightCache: Memory-Efficient, Training-Free Acceleration for Video Generation
This work addresses memory efficiency for researchers and practitioners using training-free acceleration in video generation, but it is incremental as it builds on existing cache-based methods.
The paper tackles the problem of memory surges in cache-based acceleration methods for video generation diffusion models by proposing stage-specific strategies like asynchronous cache swapping, feature chunking, and slicing latents, achieving faster inference speed and lower memory usage with acceptable quality degradation.
Training-free acceleration has emerged as an advanced research area in video generation based on diffusion models. The redundancy of latents in diffusion model inference provides a natural entry point for acceleration. In this paper, we decompose the inference process into the encoding, denoising, and decoding stages, and observe that cache-based acceleration methods often lead to substantial memory surges in the latter two stages. To address this problem, we analyze the characteristics of inference across different stages and propose stage-specific strategies for reducing memory consumption: 1) Asynchronous Cache Swapping. 2) Feature chunk. 3) Slicing latents to decode. At the same time, we ensure that the time overhead introduced by these three strategies remains lower than the acceleration gains themselves. Compared with the baseline, our approach achieves faster inference speed and lower memory usage, while maintaining quality degradation within an acceptable range. The Code is available at https://github.com/NKUShaw/LightCache .