Robust Inverse Graphics via Probabilistic Inference
This addresses the challenge of robust 3D reconstruction from corrupted images for computer vision applications, representing an incremental advance by applying Bayesian inference to a known bottleneck.
The paper tackles the problem of inferring 3D scenes from single images corrupted by rain, snow, or fog, proposing a Bayesian method called robust inverse graphics (RIG) that uses a strong scene prior and uniform corruption prior to outperform existing depth estimators and NeRF approaches, with results validated across multiple scene prior architectures.
How do we infer a 3D scene from a single image in the presence of corruptions like rain, snow or fog? Straightforward domain randomization relies on knowing the family of corruptions ahead of time. Here, we propose a Bayesian approach-dubbed robust inverse graphics (RIG)-that relies on a strong scene prior and an uninformative uniform corruption prior, making it applicable to a wide range of corruptions. Given a single image, RIG performs posterior inference jointly over the scene and the corruption. We demonstrate this idea by training a neural radiance field (NeRF) scene prior and using a secondary NeRF to represent the corruptions over which we place an uninformative prior. RIG, trained only on clean data, outperforms depth estimators and alternative NeRF approaches that perform point estimation instead of full inference. The results hold for a number of scene prior architectures based on normalizing flows and diffusion models. For the latter, we develop reconstruction-guidance with auxiliary latents (ReGAL)-a diffusion conditioning algorithm that is applicable in the presence of auxiliary latent variables such as the corruption. RIG demonstrates how scene priors can be used beyond generation tasks.