Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs
This work addresses the challenge of generating high-quality videos from text for applications in media and AI, representing an incremental improvement by enhancing dynamics awareness in diffusion models.
The paper tackles the problem of intricate temporal dynamics modeling in text-to-video synthesis, where existing diffusion models suffer from issues like action occurrence disorders and crude motions, by introducing a dynamic scene manager (Dysen) module that uses LLMs for human-level understanding and DSG representations, resulting in Dysen-VDM outperforming prior methods with significant margins, especially in complex action scenarios.
Text-to-video (T2V) synthesis has gained increasing attention in the community, in which the recently emerged diffusion models (DMs) have promisingly shown stronger performance than the past approaches. While existing state-of-the-art DMs are competent to achieve high-resolution video generation, they may largely suffer from key limitations (e.g., action occurrence disorders, crude video motions) with respect to the intricate temporal dynamics modeling, one of the crux of video synthesis. In this work, we investigate strengthening the awareness of video dynamics for DMs, for high-quality T2V generation. Inspired by human intuition, we design an innovative dynamic scene manager (dubbed as Dysen) module, which includes (step-1) extracting from input text the key actions with proper time-order arrangement, (step-2) transforming the action schedules into the dynamic scene graph (DSG) representations, and (step-3) enriching the scenes in the DSG with sufficient and reasonable details. Taking advantage of the existing powerful LLMs (e.g., ChatGPT) via in-context learning, Dysen realizes (nearly) human-level temporal dynamics understanding. Finally, the resulting video DSG with rich action scene details is encoded as fine-grained spatio-temporal features, integrated into the backbone T2V DM for video generating. Experiments on popular T2V datasets suggest that our Dysen-VDM consistently outperforms prior arts with significant margins, especially in scenarios with complex actions. Codes at https://haofei.vip/Dysen-VDM