Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models
This work addresses efficiency bottlenecks for users of VideoLLMs, offering a practical acceleration method that is incremental but compatible with existing techniques.
The paper tackles the efficiency challenges in video large language models (VideoLLMs) due to quadratic complexity from visual tokens by proposing VidCom2, a plug-and-play inference acceleration framework that adaptively compresses tokens across frames; with only 25% visual tokens, it achieves 99.6% of original performance on LLaVA-OV and reduces LLM generation latency by 70.8%.
Video large language models (VideoLLM) excel at video understanding, but face efficiency challenges due to the quadratic complexity of abundant visual tokens. Our systematic analysis of token compression methods for VideoLLMs reveals two critical issues: (i) overlooking distinctive visual signals across frames, leading to information loss; (ii) suffering from implementation constraints, causing incompatibility with modern architectures or efficient operators. To address these challenges, we distill three design principles for VideoLLM token compression and propose a plug-and-play inference acceleration framework "Video Compression Commander" (VidCom2). By quantifying each frame's uniqueness, VidCom2 adaptively adjusts compression intensity across frames, effectively preserving essential information while reducing redundancy in video sequences. Extensive experiments across various VideoLLMs and benchmarks demonstrate the superior performance and efficiency of our VidCom2. With only 25% visual tokens, VidCom2 achieves 99.6% of the original performance on LLaVA-OV while reducing 70.8% of the LLM generation latency. Notably, our Frame Compression Adjustment strategy is compatible with other token compression methods to further improve their performance. Our code is available at https://github.com/xuyang-liu16/VidCom2.