FluidNexus: 3D Fluid Reconstruction and Prediction from a Single Video
This addresses a domain-specific problem in computer vision and graphics for fluid simulation, offering a more accessible approach with potential applications in visual effects and scientific visualization.
The paper tackles the problem of reconstructing and predicting 3D fluid appearance and velocity from a single video, which previously required multi-view videos, and achieves this by synthesizing novel-view videos and integrating physics simulation.
We study reconstructing and predicting 3D fluid appearance and velocity from a single video. Current methods require multi-view videos for fluid reconstruction. We present FluidNexus, a novel framework that bridges video generation and physics simulation to tackle this task. Our key insight is to synthesize multiple novel-view videos as references for reconstruction. FluidNexus consists of two key components: (1) a novel-view video synthesizer that combines frame-wise view synthesis with video diffusion refinement for generating realistic videos, and (2) a physics-integrated particle representation coupling differentiable simulation and rendering to simultaneously facilitate 3D fluid reconstruction and prediction. To evaluate our approach, we collect two new real-world fluid datasets featuring textured backgrounds and object interactions. Our method enables dynamic novel view synthesis, future prediction, and interaction simulation from a single fluid video. Project website: https://yuegao.me/FluidNexus.