Video Patch Pruning: Efficient Video Instance Segmentation via Early Token Reduction
This work addresses efficiency for video instance segmentation, offering incremental improvements over existing patch pruning methods by extending sparsity to early layers.
The paper tackles the high computational cost of Vision Transformers in video instance segmentation by introducing Video Patch Pruning, which uses temporal prior knowledge to reduce patches by up to 60% in early layers, with a performance drop of only 0.6% on the Youtube-VIS 2021 dataset.
Vision Transformers (ViTs) have demonstrated state-ofthe-art performance in several benchmarks, yet their high computational costs hinders their practical deployment. Patch Pruning offers significant savings, but existing approaches restrict token reduction to deeper layers, leaving early-stage compression unexplored. This limits their potential for holistic efficiency. In this work, we present a novel Video Patch Pruning framework (VPP) that integrates temporal prior knowledge to enable efficient sparsity within early ViT layers. Our approach is motivated by the observation that prior features extracted from deeper layers exhibit strong foreground selectivity. Therefore we propose a fully differentiable module for temporal mapping to accurately select the most relevant patches in early network stages. Notably, the proposed method enables a patch reduction of up to 60% in dense prediction tasks, exceeding the capabilities of conventional image-based patch pruning, which typically operate around a 30% patch sparsity. VPP excels the high-sparsity regime, sustaining remarkable performance even when patch usage is reduced below 55%. Specifically, it preserves stable results with a maximal performance drop of 0.6% on the Youtube-VIS 2021 dataset.