Latent Knowledge-Guided Video Diffusion for Scientific Phenomena Generation from a Single Initial Frame
This work addresses the challenge of generating scientifically accurate videos for domains like meteorology and fluid dynamics, which is incremental as it adapts existing diffusion models with new guidance mechanisms.
The paper tackled the problem of generating videos of scientific phenomena like fluid simulations and typhoons from a single initial frame, where existing video diffusion models struggle due to domain gaps and limited data. It proposed a framework that extracts latent scientific knowledge to guide generation, achieving superior fidelity and consistency in experiments on computational fluid dynamics and real-world typhoon data.
Video diffusion models have achieved impressive results in natural scene generation, yet they struggle to generalize to scientific phenomena such as fluid simulations and meteorological processes, where underlying dynamics are governed by scientific laws. These tasks pose unique challenges, including severe domain gaps, limited training data, and the lack of descriptive language annotations. To handle this dilemma, we extracted the latent scientific phenomena knowledge and further proposed a fresh framework that teaches video diffusion models to generate scientific phenomena from a single initial frame. Particularly, static knowledge is extracted via pre-trained masked autoencoders, while dynamic knowledge is derived from pre-trained optical flow prediction. Subsequently, based on the aligned spatial relations between the CLIP vision and language encoders, the visual embeddings of scientific phenomena, guided by latent scientific phenomena knowledge, are projected to generate the pseudo-language prompt embeddings in both spatial and frequency domains. By incorporating these prompts and fine-tuning the video diffusion model, we enable the generation of videos that better adhere to scientific laws. Extensive experiments on both computational fluid dynamics simulations and real-world typhoon observations demonstrate the effectiveness of our approach, achieving superior fidelity and consistency across diverse scientific scenarios.