Language as the Medium: Multimodal Video Classification through text only
This work addresses the challenge of holistic video understanding for researchers and practitioners in multimodal machine learning by introducing a novel, incremental method that leverages existing LLMs to bridge modalities.
The authors tackled the problem of interpreting complex multimodal relationships in videos by proposing a model-agnostic approach that generates detailed textual descriptions from visual and aural modalities using LLMs like GPT-3.5 or Llama2, enabling zero-shot multimodal video classification without additional finetuning, as demonstrated on benchmarks such as UCF-101 or Kinetics.
Despite an exciting new wave of multimodal machine learning models, current approaches still struggle to interpret the complex contextual relationships between the different modalities present in videos. Going beyond existing methods that emphasize simple activities or objects, we propose a new model-agnostic approach for generating detailed textual descriptions that captures multimodal video information. Our method leverages the extensive knowledge learnt by large language models, such as GPT-3.5 or Llama2, to reason about textual descriptions of the visual and aural modalities, obtained from BLIP-2, Whisper and ImageBind. Without needing additional finetuning of video-text models or datasets, we demonstrate that available LLMs have the ability to use these multimodal textual descriptions as proxies for ``sight'' or ``hearing'' and perform zero-shot multimodal classification of videos in-context. Our evaluations on popular action recognition benchmarks, such as UCF-101 or Kinetics, show these context-rich descriptions can be successfully used in video understanding tasks. This method points towards a promising new research direction in multimodal classification, demonstrating how an interplay between textual, visual and auditory machine learning models can enable more holistic video understanding.