M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
This addresses the problem of limited progress in open vision-language models for researchers and developers by providing a large-scale, multilingual dataset, though it is incremental as it builds on existing instruction tuning methods.
The researchers tackled the scarcity of high-quality instruction datasets for vision-language models by introducing the M$^3$IT dataset, which includes 2.4 million instances and 400 instructions across 40 datasets, and they demonstrated its potential by training a model that generalizes to unseen tasks and languages.
Instruction tuning has significantly advanced large language models (LLMs) such as ChatGPT, enabling them to align with human instructions across diverse tasks. However, progress in open vision-language models (VLMs) has been limited due to the scarcity of high-quality instruction datasets. To tackle this challenge and promote research in the vision-language field, we introduce the Multi-Modal, Multilingual Instruction Tuning (M$^3$IT) dataset, designed to optimize VLM alignment with human instructions. Our M$^3$IT dataset comprises 40 carefully curated datasets, including 2.4 million instances and 400 manually written task instructions, reformatted into a vision-to-text structure. Key tasks are translated into 80 languages with an advanced translation system, ensuring broader accessibility. M$^3$IT surpasses previous datasets regarding task coverage, instruction number and instance scale. Moreover, we develop Ying-VLM, a VLM model trained on our M$^3$IT dataset, showcasing its potential to answer complex questions requiring world knowledge, generalize to unseen video tasks, and comprehend unseen instructions in Chinese. We have open-sourced the dataset to encourage further research.