ArtiFixer: Enhancing and Extending 3D Reconstruction with Auto-Regressive Diffusion Models
This addresses scalability and quality issues in generating consistent 3D reconstructions for under-observed areas, representing a strong specific gain in the domain of computer vision and graphics.
The paper tackles the problem of poor extrapolation in 3D reconstruction methods like 3D Gaussian Splatting by proposing a two-stage pipeline using a bidirectional generative model and a causal auto-regressive model, resulting in outperforming prior state-of-the-art methods by 1-3 dB PSNR on benchmark datasets.
Per-scene optimization methods such as 3D Gaussian Splatting provide state-of-the-art novel view synthesis quality but extrapolate poorly to under-observed areas. Methods that leverage generative priors to correct artifacts in these areas hold promise but currently suffer from two shortcomings. The first is scalability, as existing methods use image diffusion models or bidirectional video models that are limited in the number of views they can generate in a single pass (and thus require a costly iterative distillation process for consistency). The second is quality itself, as generators used in prior work tend to produce outputs that are inconsistent with existing scene content and fail entirely in completely unobserved regions. To solve these, we propose a two-stage pipeline that leverages two key insights. First, we train a powerful bidirectional generative model with a novel opacity mixing strategy that encourages consistency with existing observations while retaining the model's ability to extrapolate novel content in unseen areas. Second, we distill it into a causal auto-regressive model that generates hundreds of frames in a single pass. This model can directly produce novel views or serve as pseudo-supervision to improve the underlying 3D representation in a simple and highly efficient manner. We evaluate our method extensively and demonstrate that it can generate plausible reconstructions in scenarios where existing approaches fail completely. When measured on commonly benchmarked datasets, we outperform existing all existing baselines by a wide margin, exceeding prior state-of-the-art methods by 1-3 dB PSNR.