CHAI: CacHe Attention Inference for text2video
This addresses the inference bottleneck for users of text-to-video models, offering a significant speed-up without retraining, though it is incremental as it builds on existing diffusion frameworks.
The paper tackles the slow inference speed of text-to-video diffusion models by introducing CHAI, which uses cross-inference caching and Cache Attention to reduce latency while maintaining quality, achieving 1.65x to 3.35x faster generation than baseline OpenSora 1.2 with as few as 8 denoising steps.
Text-to-video diffusion models deliver impressive results but remain slow because of the sequential denoising of 3D latents. Existing approaches to speed up inference either require expensive model retraining or use heuristic-based step skipping, which struggles to maintain video quality as the number of denoising steps decreases. Our work, CHAI, aims to use cross-inference caching to reduce latency while maintaining video quality. We introduce Cache Attention as an effective method for attending to shared objects/scenes across cross-inference latents. This selective attention mechanism enables effective reuse of cached latents across semantically related prompts, yielding high cache hit rates. We show that it is possible to generate high-quality videos using Cache Attention with as few as 8 denoising steps. When integrated into the overall system, CHAI is 1.65x - 3.35x faster than baseline OpenSora 1.2 while maintaining video quality.