PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generation
This work addresses the challenge of generating physically plausible videos, which is important for video generation models, but the improvement is incremental as it combines existing RAG with video diffusion models.
PhysRAG enhances physical awareness in video generation by using retrieval-augmented generation, achieving state-of-the-art performance in visual quality and physical rule compliance on PhyGenBench and VBench benchmarks.
Developing physically aware video generation models remains a significant challenge due to the difficulty in capturing diverse physical phenomena, such as thermal dynamics, mechanics, and optics. In this work, we introduce PhysRAG, a novel pipeline that enhances physical awareness in video generation through Retrieval-Augmented Generation (RAG). To address the issue of limited high-quality data, we design a two-stage data filtering pipeline based on the WISA-80K dataset, resulting in a curated set of 7K high-quality videos for training. Furthermore, we construct a physical video database and develop a mechanism to inject physical knowledge into a video diffusion model using learnable queries. Our method achieves state-of-the-art performance in both visual quality and physical rule compliance, surpassing existing models in benchmarks such as PhyGenBench and VBench. We conduct extensive ablation studies to validate the effectiveness of our key components, including the data filtering pipeline, RAG mechanism, and method for physical information extraction. To facilitate future research, our code, data, and models are prepared for release at https://github.com/sediment1024/PhysRAG.