Rethinking Image-to-Video Adaptation: An Object-centric Perspective
This work addresses the efficiency and interpretability limitations in image-to-video adaptation for video understanding tasks, offering a novel approach that reduces computational costs while maintaining high performance.
The paper tackles the problem of efficiently adapting image models for video tasks by proposing an object-centric strategy that uses slot attention to distill frames into object tokens and models temporal changes, achieving state-of-the-art performance with only 5% of the parameters of fully finetuned models and 50% of efficient tuning methods on action recognition benchmarks.
Image-to-video adaptation seeks to efficiently adapt image models for use in the video domain. Instead of finetuning the entire image backbone, many image-to-video adaptation paradigms use lightweight adapters for temporal modeling on top of the spatial module. However, these attempts are subject to limitations in efficiency and interpretability. In this paper, we propose a novel and efficient image-to-video adaptation strategy from the object-centric perspective. Inspired by human perception, which identifies objects as key components for video understanding, we integrate a proxy task of object discovery into image-to-video transfer learning. Specifically, we adopt slot attention with learnable queries to distill each frame into a compact set of object tokens. These object-centric tokens are then processed through object-time interaction layers to model object state changes across time. Integrated with two novel object-level losses, we demonstrate the feasibility of performing efficient temporal reasoning solely on the compressed object-centric representations for video downstream tasks. Our method achieves state-of-the-art performance with fewer tunable parameters, only 5\% of fully finetuned models and 50\% of efficient tuning methods, on action recognition benchmarks. In addition, our model performs favorably in zero-shot video object segmentation without further retraining or object annotations, proving the effectiveness of object-centric video understanding.