Moment and Highlight Detection via MLLM Frame Segmentation
This addresses the problem of inefficient frame-level prediction in video analysis for researchers and practitioners, offering an incremental improvement over existing text-based and RL methods.
The paper tackles video moment and highlight detection by proposing a method that applies segmentation objectives directly to a Multimodal LLM's output tokens, using binary characters to represent frame-level probabilities. The result is strong performance with 56.74 HIT@1 for highlight detection on QVHighlights and 35.28 MAP for moment retrieval, using only 25 frames.
Detecting video moments and highlights from natural-language queries have been unified by transformer-based methods. Other works use generative Multimodal LLM (MLLM) to predict moments and/or highlights as text timestamps, utilizing its reasoning capability. While effective, text-based generation cannot provide direct gradients for frame-level predictions because the model only emits language tokens. Although recent Reinforcement Learning (RL) methods attempt to address the issue, we propose a novel approach by applying segmentation objectives directly on the LLM's output tokens. The LLM is fed with a fixed number of frames alongside a prompt that enforces it to output a sequence of continuous "0" and/or "1" characters, with one character per frame. The "0"/"1" characters benefit from the LLM's inherent language capability while also acting as background and foreground probabilities, respectively. Training employs segmentation losses on the probabilities alongside a normal causal LM loss. At inference, beam search generates sequence and logits, acting as moments and saliency scores, respectively. Despite sampling only 25 frames -- less than half of comparable methods -- our method achieved strong highlight detection (56.74 HIT@1) on QVHighlights. Additionally, our efficient method scores above the baseline (35.28 MAP) for moment retrieval. Empirically, segmentation losses provide a stable complementary learning signal even when the causal LM loss plateaus.