ETC: Extreme Token Compression via Task-aware Visual Information Distillation in VLMs
For VLM practitioners, ETC offers extreme token compression to reduce inference cost without significant task loss, though it is an incremental improvement over existing compression methods.
ETC reduces visual tokens in VLMs to a single token via task-aware variational information distillation, cutting KV-cache overhead while maintaining strong task performance on LLaVA-1.5-7B and Qwen3-VL-2B.
In Vision-Language Models (VLMs), high-resolution images produce a large number of visual tokens, resulting in high computational costs and KV-cache overhead during inference. To address this problem, we propose an Extreme Token Compression (ETC) framework that minimizes task loss when reducing the number of input tokens based on the principle of variational information distillation. Specifically, from an information-theoretic perspective, we show that minimizing task loss requires the compact representation to preserve the instruction-aware sufficient statistic of the task-relevant visual information for prediction. In practice, ETC leverages text-to-image cross-attention to weight the original visual features to approximate the latent instruction-aware predictive statistic. Moreover, ETC introduces a variational information distillation, enabling the compact representation to preserve the essential information to recover this predictive statistic. Experiments on LLaVA-1.5-7B and Qwen3-VL-2B show that ETC remains effective even under single-token compression, substantially reducing KV-cache overhead while retaining strong task performance.