Efficient Training for Human Video Generation with Entropy-Guided Prioritized Progressive Learning
This work addresses efficiency challenges for researchers and practitioners in video generation, though it is incremental as it builds on existing diffusion models.
The paper tackles the high computational cost and memory consumption in training diffusion models for human video generation by proposing Ent-Prog, an efficient training framework that achieves up to 2.2x training speedup and 2.4x GPU memory reduction while maintaining performance.
Human video generation has advanced rapidly with the development of diffusion models, but the high computational cost and substantial memory consumption associated with training these models on high-resolution, multi-frame data pose significant challenges. In this paper, we propose Entropy-Guided Prioritized Progressive Learning (Ent-Prog), an efficient training framework tailored for diffusion models on human video generation. First, we introduce Conditional Entropy Inflation (CEI) to assess the importance of different model components on the target conditional generation task, enabling prioritized training of the most critical components. Second, we introduce an adaptive progressive schedule that adaptively increases computational complexity during training by measuring the convergence efficiency. Ent-Prog reduces both training time and GPU memory consumption while maintaining model performance. Extensive experiments across three datasets, demonstrate the effectiveness of Ent-Prog, achieving up to 2.2$\times$ training speedup and 2.4$\times$ GPU memory reduction without compromising generative performance.