Lightning Fast Caching-based Parallel Denoising Prediction for Accelerating Talking Head Generation
This work addresses a practical bottleneck in generating talking head videos, offering incremental improvements tailored to this specific domain.
The paper tackles the slow inference problem in diffusion-based talking head generation by proposing a task-specific framework with caching and parallel prediction, achieving significant speed improvements while maintaining video quality.
Diffusion-based talking head models generate high-quality, photorealistic videos but suffer from slow inference, limiting practical applications. Existing acceleration methods for general diffusion models fail to exploit the temporal and spatial redundancies unique to talking head generation. In this paper, we propose a task-specific framework addressing these inefficiencies through two key innovations. First, we introduce Lightning-fast Caching-based Parallel denoising prediction (LightningCP), caching static features to bypass most model layers in inference time. We also enable parallel prediction using cached features and estimated noisy latents as inputs, efficiently bypassing sequential sampling. Second, we propose Decoupled Foreground Attention (DFA) to further accelerate attention computations, exploiting the spatial decoupling in talking head videos to restrict attention to dynamic foreground regions. Additionally, we remove reference features in certain layers to bring extra speedup. Extensive experiments demonstrate that our framework significantly improves inference speed while preserving video quality.