Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods
This work addresses a task mismatch in benchmarking for researchers developing compression techniques in MLLMs, though it is incremental as it focuses on evaluation rather than proposing a new compression method.
The paper tackles the problem of evaluating visual token compression methods for Multimodal Large Language Models, revealing that current benchmarks are noisy and that simple image downsampling often outperforms advanced methods, leading to the introduction of VTC-Bench as a denoised evaluation framework.
Recent endeavors to accelerate inference in Multimodal Large Language Models (MLLMs) have primarily focused on visual token compression. The effectiveness of these methods is typically assessed by measuring the accuracy drop on established benchmarks, comparing model performance before and after compression. However, these benchmarks are originally designed to assess the perception and reasoning capabilities of MLLMs, rather than to evaluate compression techniques. As a result, directly applying them to visual token compression introduces a task mismatch. Strikingly, our investigation reveals that simple image downsampling consistently outperforms many advanced compression methods across multiple widely used benchmarks. Through extensive experiments, we make the following observations: (i) Current benchmarks are noisy for the visual token compression task. (ii) Down-sampling is able to serve as a data filter to evaluate the difficulty of samples in the visual token compression task. Motivated by these findings, we introduce VTC-Bench, an evaluation framework that incorporates a data filtering mechanism to denoise existing benchmarks, thereby enabling fairer and more accurate assessment of visual token compression methods. All data and code are available at https://github.com/Chenfei-Liao/VTC-Bench.