Beyond Boxes: Mask-Guided Spatio-Temporal Feature Aggregation for Video Object Detection
This work addresses the problem of feature variance in video object detection for researchers and practitioners, offering a robust and method-agnostic solution with incremental improvements.
The paper tackled the challenge of exploiting temporal information in Video Object Detection by introducing a mask-guided feature aggregation method, achieving 87.9% mAP on ImageNet VID at 33 FPS and setting a new speed-accuracy benchmark.
The primary challenge in Video Object Detection (VOD) is effectively exploiting temporal information to enhance object representations. Traditional strategies, such as aggregating region proposals, often suffer from feature variance due to the inclusion of background information. We introduce a novel instance mask-based feature aggregation approach, significantly refining this process and deepening the understanding of object dynamics across video frames. We present FAIM, a new VOD method that enhances temporal Feature Aggregation by leveraging Instance Mask features. In particular, we propose the lightweight Instance Feature Extraction Module (IFEM) to learn instance mask features and the Temporal Instance Classification Aggregation Module (TICAM) to aggregate instance mask and classification features across video frames. Using YOLOX as a base detector, FAIM achieves 87.9% mAP on the ImageNet VID dataset at 33 FPS on a single 2080Ti GPU, setting a new benchmark for the speed-accuracy trade-off. Additional experiments on multiple datasets validate that our approach is robust, method-agnostic, and effective in multi-object tracking, demonstrating its broader applicability to video understanding tasks.