Triage: Hierarchical Visual Budgeting for Efficient Video Reasoning in Vision-Language Models
This addresses computational bottlenecks for researchers and practitioners using VLMs in video applications, though it is an incremental improvement as it builds on existing methods like MMR.
The paper tackles the computational inefficiency of Vision-Language Models in video processing by introducing Triage, a training-free framework that uses hierarchical visual budgeting to reduce token sequences, resulting in improved inference speed and memory usage while maintaining or exceeding baseline performance on benchmarks.
Vision-Language Models (VLMs) face significant computational challenges in video processing due to massive data redundancy, which creates prohibitively long token sequences. To address this, we introduce Triage, a training-free, plug-and-play framework that reframes video reasoning as a resource allocation problem via hierarchical visual budgeting. Its first stage, Frame-Level Budgeting, identifies keyframes by evaluating their visual dynamics and relevance, generating a strategic prior based on their importance scores. Guided by this prior, the second stage, Token-Level Budgeting, allocates tokens in two phases: it first secures high-relevance Core Tokens, followed by diverse Context Tokens selected with an efficient batched Maximal Marginal Relevance (MMR) algorithm. Extensive experiments demonstrate that Triage improves inference speed and reduces memory footprint, while maintaining or surpassing the performance of baselines and other methods on various video reasoning benchmarks.