Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models
This work addresses a key bottleneck in MLLMs for applications requiring accurate visual understanding, though it appears incremental by enhancing existing pipelines rather than introducing a new paradigm.
The paper tackles the problem of information loss in Multimodal Large Language Models (MLLMs) due to limited visual perception from CLIP-like encoders, proposing a mixture-of-experts knowledge enhancement mechanism that incorporates multi-task encoders and visual tools, resulting in improved visual perception as demonstrated in extensive experiments.
Multimodal Large Language Models (MLLMs) are experiencing rapid growth, yielding a plethora of noteworthy contributions in recent months. The prevailing trend involves adopting data-driven methodologies, wherein diverse instruction-following datasets are collected. However, a prevailing challenge persists in these approaches, specifically in relation to the limited visual perception ability, as CLIP-like encoders employed for extracting visual information from inputs. Though these encoders are pre-trained on billions of image-text pairs, they still grapple with the information loss dilemma, given that textual captions only partially capture the contents depicted in images. To address this limitation, this paper proposes to improve the visual perception ability of MLLMs through a mixture-of-experts knowledge enhancement mechanism. Specifically, we introduce a novel method that incorporates multi-task encoders and visual tools into the existing MLLMs training and inference pipeline, aiming to provide a more comprehensive and accurate summarization of visual inputs. Extensive experiments have evaluated its effectiveness of advancing MLLMs, showcasing improved visual perception achieved through the integration of visual experts.