Deformable VisTR: Spatio temporal deformable attention for video instance segmentation
This work addresses efficiency issues in transformer-based video instance segmentation, offering a practical improvement for researchers and practitioners in computer vision.
The paper tackles the high computational cost and slow convergence of VisTR for video instance segmentation by proposing Deformable VisTR, which uses a spatio-temporal deformable attention module to achieve linear computation and reduce GPU training hours by 10x while maintaining comparable performance on the Youtube-VIS benchmark.
Video instance segmentation (VIS) task requires classifying, segmenting, and tracking object instances over all frames in a video clip. Recently, VisTR has been proposed as end-to-end transformer-based VIS framework, while demonstrating state-of-the-art performance. However, VisTR is slow to converge during training, requiring around 1000 GPU hours due to the high computational cost of its transformer attention module. To improve the training efficiency, we propose Deformable VisTR, leveraging spatio-temporal deformable attention module that only attends to a small fixed set of key spatio-temporal sampling points around a reference point. This enables Deformable VisTR to achieve linear computation in the size of spatio-temporal feature maps. Moreover, it can achieve on par performance as the original VisTR with 10$\times$ less GPU training hours. We validate the effectiveness of our method on the Youtube-VIS benchmark. Code is available at https://github.com/skrya/DefVIS.