PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences
This addresses the challenge of processing raw point cloud sequences for applications like action recognition and semantic segmentation, representing an incremental improvement over existing methods.
The authors tackled the problem of modeling irregular point cloud sequences by proposing PSTNet, a deep network that uses point spatio-temporal convolution to disentangle space and time, achieving effective performance on 3D action recognition and 4D semantic segmentation datasets.
Point cloud sequences are irregular and unordered in the spatial dimension while exhibiting regularities and order in the temporal dimension. Therefore, existing grid based convolutions for conventional video processing cannot be directly applied to spatio-temporal modeling of raw point cloud sequences. In this paper, we propose a point spatio-temporal (PST) convolution to achieve informative representations of point cloud sequences. The proposed PST convolution first disentangles space and time in point cloud sequences. Then, a spatial convolution is employed to capture the local structure of points in the 3D space, and a temporal convolution is used to model the dynamics of the spatial regions along the time dimension. Furthermore, we incorporate the proposed PST convolution into a deep network, namely PSTNet, to extract features of point cloud sequences in a hierarchical manner. Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of PSTNet to model point cloud sequences.