Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM
For researchers building efficient multimodal LLMs, this work offers a method to reduce computational load while maintaining or improving performance, though it is incremental over existing Mamba-based approaches.
The paper proposes a query-based cross-modal projector that compresses visual tokens via cross-attention, eliminating the need for manual 2D scan ordering in Mamba-based multimodal LLMs. Experiments show improved performance and throughput across vision-language benchmarks.
The Transformer's quadratic complexity with input length imposes an unsustainable computational load on large language models (LLMs). In contrast, the Selective Scan Structured State-Space Model, or Mamba, addresses this computational challenge effectively. This paper explores a query-based cross-modal projector designed to bolster Mamba's efficiency for vision-language modeling by compressing visual tokens based on input through the cross-attention mechanism. This innovative projector also removes the need for manually designing the 2D scan order of original image features when converting them into an input sequence for Mamba LLM. Experimental results across various vision-language understanding benchmarks show that the proposed cross-modal projector enhances Mamba-based multimodal LLMs, boosting both performance and throughput.