Seeing Through Clutter: Structured 3D Scene Reconstruction via Iterative Object Removal
This addresses the challenge of 3D scene reconstruction for computer vision applications, particularly in complex, occluded scenes, though it is incremental as it builds on existing foundation models.
The paper tackles the problem of reconstructing structured 3D scenes from single images in cluttered environments by introducing an iterative object removal pipeline, achieving state-of-the-art robustness on 3D-Front and ADE20K datasets.
We present SeeingThroughClutter, a method for reconstructing structured 3D representations from single images by segmenting and modeling objects individually. Prior approaches rely on intermediate tasks such as semantic segmentation and depth estimation, which often underperform in complex scenes, particularly in the presence of occlusion and clutter. We address this by introducing an iterative object removal and reconstruction pipeline that decomposes complex scenes into a sequence of simpler subtasks. Using VLMs as orchestrators, foreground objects are removed one at a time via detection, segmentation, object removal, and 3D fitting. We show that removing objects allows for cleaner segmentations of subsequent objects, even in highly occluded scenes. Our method requires no task-specific training and benefits directly from ongoing advances in foundation models. We demonstrate stateof-the-art robustness on 3D-Front and ADE20K datasets. Project Page: https://rioak.github.io/seeingthroughclutter/