Controllable Weather Synthesis and Removal with Video Diffusion Models
This addresses the need for scalable and realistic weather effects in video editing for applications like film or gaming, though it is incremental as it builds on existing diffusion models.
The paper tackled the problem of generating realistic and controllable weather effects in videos, introducing WeatherWeaver, a video diffusion model that synthesizes diverse weather effects like rain, snow, fog, and clouds into input videos without 3D modeling, and it outperformed state-of-the-art methods in weather simulation and removal.
Generating realistic and controllable weather effects in videos is valuable for many applications. Physics-based weather simulation requires precise reconstructions that are hard to scale to in-the-wild videos, while current video editing often lacks realism and control. In this work, we introduce WeatherWeaver, a video diffusion model that synthesizes diverse weather effects -- including rain, snow, fog, and clouds -- directly into any input video without the need for 3D modeling. Our model provides precise control over weather effect intensity and supports blending various weather types, ensuring both realism and adaptability. To overcome the scarcity of paired training data, we propose a novel data strategy combining synthetic videos, generative image editing, and auto-labeled real-world videos. Extensive evaluations show that our method outperforms state-of-the-art methods in weather simulation and removal, providing high-quality, physically plausible, and scene-identity-preserving results over various real-world videos.