Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Models
This work addresses the accessibility issue of multimodal AI for broader research and user communities by reducing computational costs, though it is incremental as it builds on existing multimodal frameworks.
The paper tackles the high computational demands of Multimodal Large Language Models (MLLMs) by proposing Mipha, an efficient multimodal assistant based on small language models, which outperforms state-of-the-art large MLLMs like LLaVA-1.5-13B on multiple benchmarks without increasing training data.
Multimodal Large Language Models (MLLMs) have showcased impressive skills in tasks related to visual understanding and reasoning. Yet, their widespread application faces obstacles due to the high computational demands during both the training and inference phases, restricting their use to a limited audience within the research and user communities. In this paper, we investigate the design aspects of Multimodal Small Language Models (MSLMs) and propose an efficient multimodal assistant named Mipha, which is designed to create synergy among various aspects: visual representation, language models, and optimization strategies. We show that without increasing the volume of training data, our Mipha-3B outperforms the state-of-the-art large MLLMs, especially LLaVA-1.5-13B, on multiple benchmarks. Through detailed discussion, we provide insights and guidelines for developing strong MSLMs that rival the capabilities of MLLMs. Our code is available at https://github.com/zhuyiche/llava-phi.