FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression
This addresses efficiency and performance trade-offs in multi-modal AI for fine-grained visual tasks, representing an incremental advancement over existing compression methods.
The paper tackles the computational cost of high-resolution images in multi-modal large language models by proposing FocusLLaVA, a coarse-to-fine visual token compression method that removes visual redundancy, achieving improvements in both efficiency and performance across multiple datasets.
Recent advances on Multi-modal Large Language Models have demonstrated that high-resolution image input is crucial for model capabilities, especially for fine-grained tasks. However, high-resolution images lead to a quadratic increase in the number of visual tokens input into LLMs, resulting in significant computational costs. Current work develop visual token compression methods to achieve efficiency improvements, often at the expense of performance. We argue that removing visual redundancy can simultaneously improve both efficiency and performance. We build a coarse-to-fine visual token compression method, with a vision-guided sampler for compressing redundant regions with low information density, and a text-guided sampler for selecting visual tokens that are strongly correlated with the user instructions.With these two modules, the proposed FocusLLaVA achieves improvements in both efficiency and performance. We validate the effectiveness of our approach on a wide range of evaluation datasets.