SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning
This work addresses efficiency bottlenecks in video understanding for AI applications, offering a training-free method to accelerate decoding, though it is incremental as it builds on existing speculative decoding techniques.
The paper tackles the high computational cost of video large language models by introducing SpecVLM, a speculative decoding framework that prunes up to 90% of video tokens, achieving decoding speedups of up to 2.68× for LLaVA-OneVision-72B and 2.11× for Qwen2.5-VL-32B without accuracy loss.
Video large language models (Vid-LLMs) have shown strong capabilities in understanding video content. However, their reliance on dense video token representations introduces substantial memory and computational overhead in both prefilling and decoding. To mitigate the information loss of recent video token reduction methods and accelerate the decoding stage of Vid-LLMs losslessly, we introduce SpecVLM, a training-free speculative decoding (SD) framework tailored for Vid-LLMs that incorporates staged video token pruning. Building on our novel finding that the draft model's speculation exhibits low sensitivity to video token pruning, SpecVLM prunes up to 90% of video tokens to enable efficient speculation without sacrificing accuracy. To achieve this, we performs a two-stage pruning process: Stage I selects highly informative tokens guided by attention signals from the verifier (target model), while Stage II prunes remaining redundant ones in a spatially uniform manner. Extensive experiments on four video understanding benchmarks demonstrate the effectiveness and robustness of SpecVLM, which achieves up to 2.68$\times$ decoding speedup for LLaVA-OneVision-72B and 2.11$\times$ speedup for Qwen2.5-VL-32B. Code is available at https://github.com/zju-jiyicheng/SpecVLM.