Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understanding
This addresses computational and context length constraints for MLLMs in long-form video analysis, offering a more efficient and effective solution for video question answering tasks.
The paper tackles the problem of keyframe sampling for long-form video understanding in Multimodal Large Language Models (MLLMs), proposing an evidence-driven framework that maximizes conditional mutual information between frames and queries, resulting in consistent outperformance of prior methods under strict token budgets and improved training efficiency.
Multimodal Large Language Models (MLLMs) have shown strong performance on video question answering, but their application to long-form videos is constrained by limited context length and computational cost, making keyframe sampling essential. Existing approaches typically rely on semantic relevance or reinforcement learning, which either fail to capture evidential clues or suffer from inefficient combinatorial optimization. In this work, we propose an evidence-driven keyframe sampling framework grounded in information bottleneck theory. We formulate keyframe selection as maximizing the conditional mutual information between selected frames and the query, providing a principled objective that reflects each frame's contribution to answering the question. To make this objective tractable, we exploit its structure to derive a decomposed optimization that reduces subset selection to independent frame-level scoring. We further introduce a query-conditioned evidence scoring network trained with a contrastive objective to estimate evidential importance efficiently. Experiments on long-form video understanding benchmarks show that our method consistently outperforms prior sampling strategies under strict token budgets, while significantly improving training efficiency.