HierVL: Learning Hierarchical Video-Language Embeddings
This work addresses the problem of capturing broader context and intent in video-language modeling for researchers and practitioners in computer vision and multimodal AI, representing a novel method rather than an incremental improvement.
The paper tackled the limitation of existing video-language embeddings that only capture short-term associations by proposing HierVL, a hierarchical embedding that accounts for both long-term and short-term associations, achieving state-of-the-art results on tasks requiring long-term video modeling and successful transfer to multiple downstream tasks.
Video-language embeddings are a promising avenue for injecting semantics into visual representations, but existing methods capture only short-term associations between seconds-long video clips and their accompanying text. We propose HierVL, a novel hierarchical video-language embedding that simultaneously accounts for both long-term and short-term associations. As training data, we take videos accompanied by timestamped text descriptions of human actions, together with a high-level text summary of the activity throughout the long video (as are available in Ego4D). We introduce a hierarchical contrastive training objective that encourages text-visual alignment at both the clip level and video level. While the clip-level constraints use the step-by-step descriptions to capture what is happening in that instant, the video-level constraints use the summary text to capture why it is happening, i.e., the broader context for the activity and the intent of the actor. Our hierarchical scheme yields a clip representation that outperforms its single-level counterpart as well as a long-term video representation that achieves SotA results on tasks requiring long-term video modeling. HierVL successfully transfers to multiple challenging downstream tasks (in EPIC-KITCHENS-100, Charades-Ego, HowTo100M) in both zero-shot and fine-tuned settings.