Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference
For practitioners deploying multimodal LLMs, this work provides a fine-grained method to accelerate inference without discarding visual tokens or entire layers, achieving strong efficiency-accuracy trade-offs.
The paper identifies answer-silent redundancy in visual-token computation of MLLMs and proposes an operator-level skipping framework that selectively bypasses redundant attention or FFN operators. It reduces 33.7% TFLOPs on Qwen3-VL while retaining 99.5% of vanilla performance across 10 VQA benchmarks.
Multimodal large language models (MLLMs) increasingly process long visual-token sequences, increasing the overall inference computation. Existing acceleration methods usually remove visual tokens or skip visual-token updates in entire layers, but these coarse strategies may discard fine-grained evidence or suppress useful operators together with redundant ones. In this paper, we study visual-token computation from an answer-observable perspective and find that late visual-token updates can remain large while having little effect on answer-token representations. Motivated by this answer-silent redundancy, we decompose each Transformer layer into attention and FFN operators and show that useful visual computation is often operator-dominant and layer-dependent. We propose an operator-level visual-token skipping framework that preserves the full visual-token sequence while selectively bypassing redundant attention, FFN, or both. Experiments across three MLLM architectures and 10 VQA benchmarks show that our method achieves strong efficiency-accuracy trade-offs, reducing \textbf{33.7\%} TFLOPs on Qwen3-VL while retaining \textbf{99.5\%} of the vanilla model performance.