LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Models
This addresses efficiency bottlenecks in MLLMs for researchers and practitioners, though it is incremental as it builds on existing compression methods.
The paper tackled the problem of inefficient visual token compression in Multimodal Large Language Models (MLLMs) by proposing LaCo, a layer-wise compression framework that improves training efficiency by over 20% and inference throughput by over 15% while maintaining strong performance.
Existing visual token compression methods for Multimodal Large Language Models (MLLMs) predominantly operate as post-encoder modules, limiting their potential for efficiency gains. To address this limitation, we propose LaCo (Layer-wise Visual Token Compression), a novel framework that enables effective token compression within the intermediate layers of the vision encoder. LaCo introduces two core components: 1) a layer-wise pixel-shuffle mechanism that systematically merges adjacent tokens through space-to-channel transformations, and 2) a residual learning architecture with non-parametric shortcuts that preserves critical visual information during compression. Extensive experiments indicate that our LaCo outperforms all existing methods when compressing tokens in the intermediate layers of the vision encoder, demonstrating superior effectiveness. In addition, compared to external compression, our method improves training efficiency beyond 20% and inference throughput over 15% while maintaining strong performance.