FILS: Self-Supervised Video Feature Prediction In Semantic Language Space
This work addresses the challenge of semantic video understanding for computer vision applications, offering an incremental improvement over existing self-supervised approaches.
The paper tackles the problem of learning semantic video representations by proposing FILS, a self-supervised method that predicts masked video features in language space, achieving state-of-the-art results on action recognition datasets such as Epic-Kitchens and Something-SomethingV2 with ViT-Base.
This paper demonstrates a self-supervised approach for learning semantic video representations. Recent vision studies show that a masking strategy for vision and natural language supervision has contributed to developing transferable visual pretraining. Our goal is to achieve a more semantic video representation by leveraging the text related to the video content during the pretraining in a fully self-supervised manner. To this end, we present FILS, a novel self-supervised video Feature prediction In semantic Language Space (FILS). The vision model can capture valuable structured information by correctly predicting masked feature semantics in language space. It is learned using a patch-wise video-text contrastive strategy, in which the text representations act as prototypes for transforming vision features into a language space, which are then used as targets for semantically meaningful feature prediction using our masked encoder-decoder structure. FILS demonstrates remarkable transferability on downstream action recognition tasks, achieving state-of-the-art on challenging egocentric datasets, like Epic-Kitchens, Something-SomethingV2, Charades-Ego, and EGTEA, using ViT-Base. Our efficient method requires less computation and smaller batches compared to previous works.