Fourier-VLM: Compressing Vision Tokens in the Frequency Domain for Large Vision-Language Models
This addresses efficiency issues for users of large vision-language models, offering a practical solution with significant speed-ups, though it is incremental as it builds on existing VLM architectures.
The paper tackles the high computational overhead and inference latency in Vision-Language Models (VLMs) caused by many vision tokens by proposing Fourier-VLM, which compresses visual representations in the frequency domain using a low-pass filter based on the Discrete Cosine Transform, achieving up to 83.8% reduction in inference FLOPs and 31.2% faster generation speed while maintaining competitive performance.
Vision-Language Models (VLMs) typically replace the predefined image placeholder token (<image>) in textual instructions with visual features from an image encoder, forming the input to a backbone Large Language Model (LLM). However, the large number of vision tokens significantly increases the context length, leading to high computational overhead and inference latency. While previous efforts mitigate this by selecting only important visual features or leveraging learnable queries to reduce token count, they often compromise performance or introduce substantial extra costs. In response, we propose Fourier-VLM, a simple yet efficient method that compresses visual representations in the frequency domain. Our approach is motivated by the observation that vision features output from the vision encoder exhibit concentrated energy in low-frequency components. Leveraging this, we apply a low-pass filter to the vision features using a two-dimensional Discrete Cosine Transform (DCT). Notably, the DCT is efficiently computed via the Fast Fourier Transform (FFT) operator with a time complexity of $\mathcal{O}(n\log n)$, minimizing the extra computational cost while introducing no additional parameters. Extensive experiments across various image-based benchmarks demonstrate that Fourier-VLM achieves competitive performance with strong generalizability across both LLaVA and Qwen-VL architectures. Crucially, it reduce inference FLOPs by up to 83.8% and boots generation speed by 31.2% compared to LLaVA-v1.5, highlighting the superior efficiency and practicality.