Generalizing Single-View 3D Shape Retrieval to Occlusions and Unseen Objects
This addresses the challenge of making 3D shape retrieval more practical for real-world applications like robotics or AR, but it is incremental as it builds on existing datasets and methods.
The paper tackled the problem of single-view 3D shape retrieval under realistic occlusions and unseen objects, finding that prior methods degrade with occlusion, and by pretraining on synthetic occluded data and finetuning on real data, it significantly outperforms previous models with robustness to unseen shapes and objects.
Single-view 3D shape retrieval is a challenging task that is increasingly important with the growth of available 3D data. Prior work that has studied this task has not focused on evaluating how realistic occlusions impact performance, and how shape retrieval methods generalize to scenarios where either the target 3D shape database contains unseen shapes, or the input image contains unseen objects. In this paper, we systematically evaluate single-view 3D shape retrieval along three different axes: the presence of object occlusions and truncations, generalization to unseen 3D shape data, and generalization to unseen objects in the input images. We standardize two existing datasets of real images and propose a dataset generation pipeline to produce a synthetic dataset of scenes with multiple objects exhibiting realistic occlusions. Our experiments show that training on occlusion-free data as was commonly done in prior work leads to significant performance degradation for inputs with occlusion. We find that that by first pretraining on our synthetic dataset with occlusions and then finetuning on real data, we can significantly outperform models from prior work and demonstrate robustness to both unseen 3D shapes and unseen objects.