SmokeSeer: 3D Gaussian Splatting for Smoke Removal and Scene Reconstruction
This addresses the issue of smoke obscuring scenes for applications like surveillance or rescue, but it is incremental as it builds on 3D Gaussian splatting with multi-modal fusion.
The paper tackles the problem of smoke degrading image quality and visibility by introducing SmokeSeer, a method for simultaneous 3D scene reconstruction and smoke removal from multi-view video using thermal and RGB images, which handles a broad range of smoke densities and adapts to temporal variations, validated on synthetic data and a new real-world dataset.
Smoke in real-world scenes can severely degrade the quality of images and hamper visibility. Recent methods for image restoration either rely on data-driven priors that are susceptible to hallucinations, or are limited to static low-density smoke. We introduce SmokeSeer, a method for simultaneous 3D scene reconstruction and smoke removal from a video capturing multiple views of a scene. Our method uses thermal and RGB images, leveraging the fact that the reduced scattering in thermal images enables us to see through the smoke. We build upon 3D Gaussian splatting to fuse information from the two image modalities, and decompose the scene explicitly into smoke and non-smoke components. Unlike prior approaches, SmokeSeer handles a broad range of smoke densities and can adapt to temporally varying smoke. We validate our approach on synthetic data and introduce a real-world multi-view smoke dataset with RGB and thermal images. We provide open-source code and data at the project website.