LLAVADI: What Matters For Multimodal Large Language Models Distillation
This work addresses deployment challenges for MLLMs by enabling efficient small-scale models, though it is incremental as it builds on existing distillation methods.
The paper tackles the problem of reducing memory and computational demands of Multimodal Large Language Models (MLLMs) by focusing on knowledge distillation to train small-scale models, showing that a 2.7B model can perform on par with larger 7B or 13B models.
The recent surge in Multimodal Large Language Models (MLLMs) has showcased their remarkable potential for achieving generalized intelligence by integrating visual understanding into Large Language Models.Nevertheless, the sheer model size of MLLMs leads to substantial memory and computational demands that hinder their widespread deployment. In this work, we do not propose a new efficient model structure or train small-scale MLLMs from scratch. Instead, we focus on what matters for training small-scale MLLMs through knowledge distillation, which is the first step from the multimodal distillation perspective. Our extensive studies involve training strategies, model choices, and distillation algorithms in the knowledge distillation process. These results show that joint alignment for both tokens and logit alignment plays critical roles in teacher-student frameworks. In addition, we draw a series of intriguing observations from this study. By evaluating different benchmarks and proper strategy, even a 2.7B small-scale model can perform on par with larger models with 7B or 13B parameters. Our code and models will be publicly available for further research.