PEEK: Picking Essential frames via Efficient Knowledge distillation
This work provides a more efficient and accurate frame selection method for researchers and practitioners working with video-language models, particularly for video captioning tasks where computational resources or frame budgets are limited.
The paper addresses the bottleneck of limited frame processing in video-language models for video captioning by introducing PEEK, an efficient dynamic frame sampling method. PEEK distills frame relevance rankings from a teacher model into a lightweight temporal model, outperforming state-of-the-art methods on ActivityNet Captions and MSR-VTT, especially when selecting one or two frames. It achieves the best CIDEr for most frame budgets and adds only 5.2% to captioning time, significantly less than other adaptive baselines.
Video-language models can process only a limited number of frames, making frame selection a key bottleneck for efficient video captioning. Most captioning pipelines still rely on uniform sampling, which is computationally cheap but agnostic to visual content. Adaptive frame sampling has recently emerged as a promising approach for selecting the most informative frames from a video; however, existing methods remain computationally expensive. We introduce PEEK, an efficient dynamic frame sampling method that distills caption-conditioned frame relevance rankings from a stronger teacher model into a lightweight temporal model that operates only on visual content. We find that, overall, on ActivityNet Captions and MSR-VTT, our method outperforms state-of-the-art methods across all evaluated downstream vision language models, especially when only one or two frames are selected for captioning, obtaining the best CIDEr for most frame budgets. On ActivityNet Captions, PEEK is particularly strong, winning 14 out of 16 configurations. Zero-shot evaluation on MSR-VTT shows that our model transfers best at low frame budgets, while results at four and eight frames are more mixed as temporal coverage and visual diversity become increasingly competitive. Compared with recent adaptive baselines, PEEK is both more accurate in the low-budget regime and more efficient: it adds only $5.2\%$ to the captioning time, compared with $65.4\%$ for CSTA and $211.9\%$ for MaxInfo. We release our code and pre-trained checkpoint at https://github.com/momentslab/peek.