VTD-CLIP: Video-to-Text Discretization via Prompting CLIP
This work addresses video recognition challenges for researchers and practitioners by enhancing interpretability and generalization, though it is incremental as it builds on existing CLIP-based methods.
The paper tackles the problem of limited interpretability and poor generalization in video recognition by proposing a video-to-text discretization framework that transforms temporal visual data into textual tokens using a visual codebook derived from CLIP's text encoder, achieving competitive improvements on benchmarks like HMDB-51, UCF-101, SSv2, and Kinetics-400.
Vision-language models bridge visual and linguistic understanding and have proven to be powerful for video recognition tasks. Existing approaches primarily rely on parameter-efficient fine-tuning of image-text pre-trained models, yet they often suffer from limited interpretability and poor generalization due to inadequate temporal modeling. To address these, we propose a simple yet effective video-to-text discretization framework. Our method repurposes the frozen text encoder to construct a visual codebook from video class labels due to the many-to-one contrastive alignment between visual and textual embeddings in multimodal pretraining. This codebook effectively transforms temporal visual data into textual tokens via feature lookups and offers interpretable video representations through explicit video modeling. Then, to enhance robustness against irrelevant or noisy frames, we introduce a confidence-aware fusion module that dynamically weights keyframes by assessing their semantic relevance via the codebook. Furthermore, our method incorporates learnable text prompts to conduct adaptive codebook updates. Extensive experiments on HMDB-51, UCF-101, SSv2, and Kinetics-400 have validated the superiority of our approach, achieving more competitive improvements over state-of-the-art methods. The code will be publicly available at https://github.com/isxinxin/VTD-CLIP.