PointConvFormer: Revenge of the Point-based Convolution
This work addresses the need for efficient and accurate point-level tasks such as segmentation and scene flow estimation in computer vision, representing an incremental improvement by hybridizing existing techniques.
The authors tackled the problem of improving point cloud processing by introducing PointConvFormer, a novel building block that combines point convolution and Transformers, resulting in a better accuracy-speed tradeoff compared to existing methods like classic convolutions and regular transformers.
We introduce PointConvFormer, a novel building block for point cloud based deep network architectures. Inspired by generalization theory, PointConvFormer combines ideas from point convolution, where filter weights are only based on relative position, and Transformers which utilize feature-based attention. In PointConvFormer, attention computed from feature difference between points in the neighborhood is used to modify the convolutional weights at each point. Hence, we preserved the invariances from point convolution, whereas attention helps to select relevant points in the neighborhood for convolution. PointConvFormer is suitable for multiple tasks that require details at the point level, such as segmentation and scene flow estimation tasks. We experiment on both tasks with multiple datasets including ScanNet, SemanticKitti, FlyingThings3D and KITTI. Our results show that PointConvFormer offers a better accuracy-speed tradeoff than classic convolutions, regular transformers, and voxelized sparse convolution approaches. Visualizations show that PointConvFormer performs similarly to convolution on flat areas, whereas the neighborhood selection effect is stronger on object boundaries, showing that it has got the best of both worlds.