Parallel Vision Token Scheduling for Fast and Accurate Multimodal LMMs Inference
This addresses inference speed bottlenecks for users of MLLMs, offering a practical improvement with minimal performance trade-offs, though it is incremental as it builds on existing architectures.
The paper tackles the high inference latency in multimodal large language models (MLLMs) caused by many visual tokens, by introducing ParVTS, a training-free scheduling framework that partitions and processes visual tokens in parallel, achieving up to 88.9% token pruning with minimal accuracy loss, 1.77x speedup, and 70% FLOPs reduction.
Multimodal large language models (MLLMs) deliver impressive vision-language reasoning but suffer steep inference latency because self-attention scales quadratically with sequence length and thousands of visual tokens contributed by high-resolution images. Naively pruning less-informative visual tokens reduces this burden, yet indiscriminate removal can strip away contextual cues essential for background or fine-grained questions, undermining accuracy. In this paper, we present ParVTS (Parallel Vision Token Scheduling), a training-free scheduling framework that partitions visual tokens into subject and non-subject groups, processes them in parallel to transfer their semantics into question tokens, and discards the non-subject path mid-inference to reduce computation. This scheduling reduces computational complexity, requires no heuristics or additional modules, and is compatible with diverse existing MLLM architectures. Experiments across multiple MLLM backbones show that ParVTS prunes up to 88.9% of visual tokens with minimal performance drop, achieving 1.77x speedup and 70% FLOPs reduction.