VSFormer: Mining Correlations in Flexible View Set for Multi-view 3D Shape Understanding
This work addresses the need for more effective multi-view 3D shape recognition and retrieval, with incremental improvements in method design for this domain-specific task.
The paper tackled the problem of limited flexibility and indirect correlation learning in view-based methods for 3D shape understanding by proposing VSFormer, a Transformer model that explicitly captures correlations in a flexible view set, achieving state-of-the-art results on datasets like ModelNet40, ScanObjectNN, RGBD, and SHREC'17.
View-based methods have demonstrated promising performance in 3D shape understanding. However, they tend to make strong assumptions about the relations between views or learn the multi-view correlations indirectly, which limits the flexibility of exploring inter-view correlations and the effectiveness of target tasks. To overcome the above problems, this paper investigates flexible organization and explicit correlation learning for multiple views. In particular, we propose to incorporate different views of a 3D shape into a permutation-invariant set, referred to as \emph{View Set}, which removes rigid relation assumptions and facilitates adequate information exchange and fusion among views. Based on that, we devise a nimble Transformer model, named \emph{VSFormer}, to explicitly capture pairwise and higher-order correlations of all elements in the set. Meanwhile, we theoretically reveal a natural correspondence between the Cartesian product of a view set and the correlation matrix in the attention mechanism, which supports our model design. Comprehensive experiments suggest that VSFormer has better flexibility, efficient inference efficiency and superior performance. Notably, VSFormer reaches state-of-the-art results on various 3d recognition datasets, including ModelNet40, ScanObjectNN and RGBD. It also establishes new records on the SHREC'17 retrieval benchmark. The code and datasets are available at \url{https://github.com/auniquesun/VSFormer}.