Benefit from Reference: Retrieval-Augmented Cross-modal Point Cloud Completion
This work addresses a domain-specific problem in 3D vision for applications like robotics or AR/VR, with incremental improvements over existing cross-modal methods.
The paper tackles the challenge of completing 3D structures from incomplete point clouds by proposing a retrieval-augmented framework that learns structural priors from similar reference samples, achieving effectiveness in generating fine-grained point clouds and generalization to sparse data and unseen categories.
Completing the whole 3D structure based on an incomplete point cloud is a challenging task, particularly when the residual point cloud lacks typical structural characteristics. Recent methods based on cross-modal learning attempt to introduce instance images to aid the structure feature learning. However, they still focus on each particular input class, limiting their generation abilities. In this work, we propose a novel retrieval-augmented point cloud completion framework. The core idea is to incorporate cross-modal retrieval into completion task to learn structural prior information from similar reference samples. Specifically, we design a Structural Shared Feature Encoder (SSFE) to jointly extract cross-modal features and reconstruct reference features as priors. Benefiting from a dual-channel control gate in the encoder, relevant structural features in the reference sample are enhanced and irrelevant information interference is suppressed. In addition, we propose a Progressive Retrieval-Augmented Generator (PRAG) that employs a hierarchical feature fusion mechanism to integrate reference prior information with input features from global to local. Through extensive evaluations on multiple datasets and real-world scenes, our method shows its effectiveness in generating fine-grained point clouds, as well as its generalization capability in handling sparse data and unseen categories.