DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models
This addresses a specific bottleneck in MLLMs for vision-language tasks, offering an incremental improvement by balancing detailed and holistic analysis.
The paper tackles the problem of multi-modal large language models (MLLMs) having deficiencies in detailed vision-language tasks due to singular focus on predefined resolutions, and introduces DualFocus to integrate macro and micro perspectives, resulting in reduced hallucination and improved performance across benchmarks.
We present DualFocus, a novel framework for integrating macro and micro perspectives within multi-modal large language models (MLLMs) to enhance vision-language task performance. Current MLLMs typically singularly focus on inputs at a predefined resolution, resulting in deficiencies in detailed questions involving local regions. We introduced a DualFocus mechanism where the model concentrates on the image from a macro perspective, responses to the question, and identifies suitable sub-regions to zoom in for subsequent micro perspective analysis. Via the integration of answers from both macro and micro perspectives, the model is adept at addressing tasks that encompass global, detailed, and combined considerations. To endows the DualFocus mechanism in MLLMs, we curated a tailored dataset derived from the Visual Genome (VG) and adapted it to align with the training regimen of DualFocus. Through comparative studies across different model sizes and benchmarks, we demonstrate DualFocus's superiority in balancing detailed examination with holistic insight, significantly reducing hallucination instances in MLLMs and improving their performance in various vision-language tasks.