Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices
This work addresses a specific bottleneck in MLLMs for researchers and practitioners, offering incremental improvements in visual feature fusion methods.
The paper tackles the problem of suboptimal integration of multi-layer visual features in multimodal large language models by systematically investigating layer selection and fusion strategies, finding that combining features from multiple stages improves generalization while same-stage features degrade performance, and direct input-stage fusion yields superior and stable results.
Multimodal Large Language Models (MLLMs) have made significant advancements in recent years, with visual features playing an increasingly critical role in enhancing model performance. However, the integration of multi-layer visual features in MLLMs remains underexplored, particularly with regard to optimal layer selection and fusion strategies. Existing methods often rely on arbitrary design choices, leading to suboptimal outcomes. In this paper, we systematically investigate two core aspects of multi-layer visual feature fusion: (1) selecting the most effective visual layers and (2) identifying the best fusion approach with the language model. Our experiments reveal that while combining visual features from multiple stages improves generalization, incorporating additional features from the same stage typically leads to diminished performance. Furthermore, we find that direct fusion of multi-layer visual features at the input stage consistently yields superior and more stable performance across various configurations. We make all our code publicly available: https://github.com/EIT-NLP/Layer_Select_Fuse_for_MLLM.