Benefit from reference: Retrieval-augmented cross-modal point cloud completion

Completing the whole 3D structure based on an incomplete point cloud is a challenging task, particularly when the residual point cloud lacks typical structural characteristics. Recent methods based on cross-modal inputs attempt to introduce the 2D image to facilitate the learning of global and local features. However, the inherent differences between different modalities impact the effectiveness for unseen classes or varied structural patterns. In reality, similar objects can serve as important references for completing results. Therefore, we propose a novel retrieval-augmented point cloud completion framework. The core idea is to integrate cross-modal retrieval into the completion task, thereby acquiring prior information from similar 3D samples. Specifically, we design a Structural Shared Feature Encoder (SSFE) that jointly extracts cross-modal input features and obtains structural prior information from the reference point cloud. The final feature representation in the encoder benefits from a dual-channel control gate, which both enhances relevant structural features and reduces interference from irrelevant information. In addition, we propose a Progressive Retrieval-Augmented Generator (PRAG) that employs a hierarchical feature fusion mechanism to integrate reference prior information, generating complete point clouds from global to local levels. Through extensive evaluations on multiple datasets and real-world scenes, our method shows its effectiveness in generating fine-grained point clouds, as well as its generalization capability in handling sparse data and unseen categories. Without introducing any extra inputs, our approach outperforms the state-of-the-art by reducing the Chamfer Distance by 0.223 and improving the F1-score by 5.3% on the Shapenet-ViPC dataset.