Language-based Action Concept Spaces Improve Video Self-Supervised Learning
This work addresses the problem of video representation learning for action recognition, but it is incremental as it builds on existing CLIP models with modifications.
The paper tackles adapting image-based contrastive language-image pre-training (CLIP) models to video domains with minimal supervision, achieving improved zero-shot and linear probing performance on three action recognition benchmarks.
Recent contrastive language image pre-training has led to learning highly transferable and robust image representations. However, adapting these models to video domains with minimal supervision remains an open problem. We explore a simple step in that direction, using language tied self-supervised learning to adapt an image CLIP model to the video domain. A backbone modified for temporal modeling is trained under self-distillation settings with train objectives operating in an action concept space. Feature vectors of various action concepts extracted from a language encoder using relevant textual prompts construct this space. We introduce two train objectives, concept distillation and concept alignment, that retain generality of original representations while enforcing relations between actions and their attributes. Our approach improves zero-shot and linear probing performance on three action recognition benchmarks.