VidCLearn: A Continual Learning Approach for Text-to-Video Generation
This addresses the challenge of updating text-to-video generation models efficiently for AI practitioners, though it is incremental as it builds on existing diffusion-based methods.
The paper tackles the problem of text-to-video generation models being unable to incorporate new data without full retraining by proposing VidCLearn, a continual learning framework that uses a student-teacher architecture with generative replay, achieving superior visual quality, semantic alignment, and temporal coherence compared to baselines.
Text-to-video generation is an emerging field in generative AI, enabling the creation of realistic, semantically accurate videos from text prompts. While current models achieve impressive visual quality and alignment with input text, they typically rely on static knowledge, making it difficult to incorporate new data without retraining from scratch. To address this limitation, we propose VidCLearn, a continual learning framework for diffusion-based text-to-video generation. VidCLearn features a student-teacher architecture where the student model is incrementally updated with new text-video pairs, and the teacher model helps preserve previously learned knowledge through generative replay. Additionally, we introduce a novel temporal consistency loss to enhance motion smoothness and a video retrieval module to provide structural guidance at inference. Our architecture is also designed to be more computationally efficient than existing models while retaining satisfactory generation performance. Experimental results show VidCLearn's superiority over baseline methods in terms of visual quality, semantic alignment, and temporal coherence.