Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models
This work addresses a specific bottleneck in large multimodal models for researchers and practitioners by improving high-resolution image processing and scene understanding, though it is incremental as it builds on existing vision encoders and adapts methods for enhanced capabilities.
The paper tackles the challenge of large multimodal models struggling with high-resolution input and detailed scene understanding by introducing Monkey, which processes images in uniform patches with individual adapters to handle higher resolutions and employs multi-level description generation, achieving superior performance on 18 datasets in tasks like image captioning and visual question answering compared to existing models.
Large Multimodal Models (LMMs) have shown promise in vision-language tasks but struggle with high-resolution input and detailed scene understanding. Addressing these challenges, we introduce Monkey to enhance LMM capabilities. Firstly, Monkey processes input images by dividing them into uniform patches, each matching the size (e.g., 448x448) used in the original training of the well-trained vision encoder. Equipped with individual adapter for each patch, Monkey can handle higher resolutions up to 1344x896 pixels, enabling the detailed capture of complex visual information. Secondly, it employs a multi-level description generation method, enriching the context for scene-object associations. This two-part strategy ensures more effective learning from generated data: the higher resolution allows for a more detailed capture of visuals, which in turn enhances the effectiveness of comprehensive descriptions. Extensive ablative results validate the effectiveness of our designs. Additionally, experiments on 18 datasets further demonstrate that Monkey surpasses existing LMMs in many tasks like Image Captioning and various Visual Question Answering formats. Specially, in qualitative tests focused on dense text question answering, Monkey has exhibited encouraging results compared with GPT4V. Code is available at https://github.com/Yuliang-Liu/Monkey.