FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding
This addresses a critical bottleneck for researchers and practitioners using video-LMMs by enabling faster processing of long videos with near-optimal performance, though it is incremental as it builds on existing compression techniques.
The paper tackles the scalability issue in long video understanding caused by excessive visual tokens by proposing FLoC, a training-free compression framework that selects a compact subset of tokens using a facility location function, achieving significant efficiency gains and outperforming recent methods on benchmarks like Video-MME, MLVU, and LongVideoBench.
Recent studies in long video understanding have harnessed the advanced visual-language reasoning capabilities of Large Multimodal Models (LMMs), driving the evolution of video-LMMs specialized for processing extended video sequences. However, the scalability of these models is severely limited by the overwhelming volume of visual tokens generated from extended video sequences. To address this challenge, this paper proposes FLoC, an efficient visual token compression framework based on the facility location function, a principled approach that swiftly selects a compact yet highly representative and diverse subset of visual tokens within a predefined budget on the number of visual tokens. By integrating the lazy greedy algorithm, our method achieves remarkable efficiency gains by swiftly selecting a compact subset of tokens, drastically reducing the number of visual tokens while guaranteeing near-optimal performance. Notably, our approach is training-free, model-agnostic, and query-agnostic, providing a versatile solution that seamlessly integrates with diverse video-LLMs and existing workflows. Extensive evaluations on large-scale benchmarks, such as Video-MME, MLVU, and LongVideoBench, demonstrate that our framework consistently surpasses recent compression techniques, highlighting not only its effectiveness and robustness in addressing the critical challenges of long video understanding, but also its efficiency in processing speed.