Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning
This addresses the efficiency bottleneck in visual instruction tuning for large language models, representing a strong incremental improvement.
The paper tackles the trade-off between accuracy and efficiency in vision-to-language projection for visual instruction tuning by introducing LLaVA-Meteor with a Top-Down Compression paradigm, which reduces visual tokens by 75-95% while achieving comparable or superior performance on 12 benchmarks.
Visual instruction tuning aims to enable large language models to comprehend the visual world, with a pivotal challenge lying in establishing an effective vision-to-language projection. However, existing methods often grapple with the intractable trade-off between accuracy and efficiency. In this paper, we present LLaVA-Meteor, a novel approach designed to break this deadlock, equipped with a novel Top-Down Compression paradigm that strategically compresses visual tokens without compromising core information. Specifically, we construct a trainable Flash Global Fusion module based on efficient selective state space operators, which aligns the feature space while enabling each token to perceive holistic visual context and instruction preference at low cost. Furthermore, a local-to-single scanning manner is employed to effectively capture local dependencies, thereby enhancing the model's capability in vision modeling. To alleviate computational overhead, we explore a Visual-Native Selection mechanism that independently assesses token significance by both the visual and native experts, followed by aggregation to retain the most critical subset. Extensive experiments show that our approach reduces visual tokens by 75--95% while achieving comparable or superior performance across 12 benchmarks, significantly improving efficiency.