ViFu: Multiple 360$^\circ$ Objects Reconstruction with Clean Background via Visible Part Fusion
This addresses the limitation of neural radiance fields in modeling occluded parts for scene editing and synthetic data generation, though it is incremental as it builds on existing methods with a novel fusion approach.
The paper tackles the problem of reconstructing multiple 360-degree objects and a clean background from scenes observed at different timestamps by fusing visible parts across scenes, achieving occlusion-free rendering as demonstrated in experiments on synthetic and real datasets.
In this paper, we propose a method to segment and recover a static, clean background and multiple 360$^\circ$ objects from observations of scenes at different timestamps. Recent works have used neural radiance fields to model 3D scenes and improved the quality of novel view synthesis, while few studies have focused on modeling the invisible or occluded parts of the training images. These under-reconstruction parts constrain both scene editing and rendering view selection, thereby limiting their utility for synthetic data generation for downstream tasks. Our basic idea is that, by observing the same set of objects in various arrangement, so that parts that are invisible in one scene may become visible in others. By fusing the visible parts from each scene, occlusion-free rendering of both background and foreground objects can be achieved. We decompose the multi-scene fusion task into two main components: (1) objects/background segmentation and alignment, where we leverage point cloud-based methods tailored to our novel problem formulation; (2) radiance fields fusion, where we introduce visibility field to quantify the visible information of radiance fields, and propose visibility-aware rendering for the fusion of series of scenes, ultimately obtaining clean background and 360$^\circ$ object rendering. Comprehensive experiments were conducted on synthetic and real datasets, and the results demonstrate the effectiveness of our method.