Error-driven Data-efficient Large Multimodal Model Tuning
This addresses the challenge of data-efficient adaptation for LMMs in downstream applications, offering a practical solution for scenarios where task-specific data is scarce or costly, though it is incremental as it builds on existing tuning methods.
The paper tackles the problem of fine-tuning large multimodal models (LMMs) for new tasks without task-specific training data by proposing an error-driven framework that uses a teacher model to identify student model errors and retrieve targeted samples from existing datasets, resulting in an average performance boost of 7.01% across seven tasks.
Large Multimodal Models (LMMs) have demonstrated impressive performance across numerous academic benchmarks. However, fine-tuning still remains essential to achieve satisfactory performance on downstream tasks, while the task-specific tuning samples are usually not readily available or expensive and time-consuming to obtain. To address this, we propose an error-driven data-efficient tuning framework that aims to efficiently adapt generic LMMs to newly emerging tasks without requiring any task-specific training samples. In our approach, a generic LMM, acting as a student model, is first evaluated on a small validation set of the target task, and then a more powerful model, acting as a teacher model, identifies the erroneous steps within the student model's reasoning steps and analyzes its capability gaps from fully addressing the target task. Based on these gaps, targeted training samples are further retrieved from existing task-agnostic datasets to tune the student model and tailor it to the target task. We perform extensive experiments across three different training data scales and seven tasks, demonstrating that our training paradigm significantly and efficiently improves LMM's performance on downstream tasks, achieving an average performance boost of 7.01%.