StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understanding
This addresses efficiency issues for applications like public surveillance and AI glasses, but is incremental as it builds on existing pruning methods.
The paper tackled the challenge of high GPU memory usage and computational latency in online video understanding with Multimodal Large Language Models by proposing token pruning to reduce context length, achieving up to 4% accuracy improvement with less than 1ms pruning latency.
Online video understanding is essential for applications like public surveillance and AI glasses. However, applying Multimodal Large Language Models (MLLMs) to this domain is challenging due to the large number of video frames, resulting in high GPU memory usage and computational latency. To address these challenges, we propose token pruning as a means to reduce context length while retaining critical information. Specifically, we introduce a novel redundancy metric, Maximum Similarity to Spatially Adjacent Video Tokens (MSSAVT), which accounts for both token similarity and spatial position. To mitigate the bidirectional dependency between pruning and redundancy, we further design a masked pruning strategy that ensures only mutually unadjacent tokens are pruned. We also integrate an existing temporal redundancy-based pruning method to eliminate temporal redundancy of the video modality. Experimental results on multiple online and offline video understanding benchmarks demonstrate that our method significantly improves the accuracy (i.e., by 4\% at most) while incurring a negligible pruning latency (i.e., less than 1ms). Our full implementation will be made publicly available.