Spatial-Temporal Transformer for 3D Point Cloud Sequences
This addresses the challenge of processing dynamic 3D point clouds for applications such as autonomous driving or robotics, though it appears incremental as it builds on transformer architectures.
The authors tackled the problem of learning spatial-temporal representations from dynamic 3D point cloud sequences for tasks like 4D semantic segmentation and 3D action recognition, and their PST2 framework outperformed existing methods on all three benchmarks tested.
Effective learning of spatial-temporal information within a point cloud sequence is highly important for many down-stream tasks such as 4D semantic segmentation and 3D action recognition. In this paper, we propose a novel framework named Point Spatial-Temporal Transformer (PST2) to learn spatial-temporal representations from dynamic 3D point cloud sequences. Our PST2 consists of two major modules: a Spatio-Temporal Self-Attention (STSA) module and a Resolution Embedding (RE) module. Our STSA module is introduced to capture the spatial-temporal context information across adjacent frames, while the RE module is proposed to aggregate features across neighbors to enhance the resolution of feature maps. We test the effectiveness our PST2 with two different tasks on point cloud sequences, i.e., 4D semantic segmentation and 3D action recognition. Extensive experiments on three benchmarks show that our PST2 outperforms existing methods on all datasets. The effectiveness of our STSA and RE modules have also been justified with ablation experiments.