Automatic Detection of Intro and Credits in Video using CLIP and Multihead Attention
This work addresses the labor-intensive and error-prone task of manual annotation for content segmentation in videos, with applications in automated indexing and recommendation systems, though it is incremental as it builds on existing CLIP and attention models.
The paper tackled the problem of automatically detecting intro and credits transitions in videos by developing a deep learning-based sequence-to-sequence classification method, achieving an F1-score of 91.0% and real-time inference speeds up to 107 FPS on GPUs.
Detecting transitions between intro/credits and main content in videos is a crucial task for content segmentation, indexing, and recommendation systems. Manual annotation of such transitions is labor-intensive and error-prone, while heuristic-based methods often fail to generalize across diverse video styles. In this work, we introduce a deep learning-based approach that formulates the problem as a sequence-to-sequence classification task, where each second of a video is labeled as either "intro" or "film." Our method extracts frames at a fixed rate of 1 FPS, encodes them using CLIP (Contrastive Language-Image Pretraining), and processes the resulting feature representations with a multihead attention model incorporating learned positional encoding. The system achieves an F1-score of 91.0%, Precision of 89.0%, and Recall of 97.0% on the test set, and is optimized for real-time inference, achieving 11.5 FPS on CPU and 107 FPS on high-end GPUs. This approach has practical applications in automated content indexing, highlight detection, and video summarization. Future work will explore multimodal learning, incorporating audio features and subtitles to further enhance detection accuracy.