HIRE: Lightweight High-Resolution Image Feature Enrichment for Multimodal LLMs
This work addresses efficiency issues for researchers and practitioners using multimodal LLMs, though it is incremental as it builds on existing feature upsampling ideas.
The paper tackles the high computational cost of integrating high-resolution image features in multimodal LLMs by proposing a lightweight feature enricher, achieving competitive results with up to 1.5x savings in FLOPs.
The integration of high-resolution image features in modern multimodal large language models has demonstrated significant improvements in fine-grained visual understanding tasks, achieving high performance across multiple benchmarks. Since these features are obtained from large image encoders like ViT, they come with a significant increase in computational costs due to multiple calls to these encoders. In this work, we first develop an intuition for feature upsampling as a natural extension of high-resolution feature generation. Through extensive experiments and ablations, we demonstrate how a shallow feature enricher can achieve competitive results with tremendous reductions in training and inference time as well as computational cost, with upto 1.5x saving in FLOPs.