Improve MLLM Benchmark Efficiency through Interview
This work addresses efficiency issues in benchmarking MLLMs for researchers and developers, but it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of resource-intensive and time-consuming full-coverage Q&A testing for Multimodal Large Language Models (MLLMs) by proposing the MLLM Interview (MITV) strategy, which uses difficulty-labeled questions to quickly assess model performance with fewer queries, achieving faster evaluation capabilities as shown in experiments.
The rapid development of Multimodal Large Language Models (MLLM) has led to a wide range of MLLM applications, and a number of benchmark datasets have sprung up in order to assess MLLM abilities. However, full-coverage Q&A testing on large-scale data is resource-intensive and time-consuming. To address this issue, we propose the MLLM Interview (MITV) strategy, which aims to quickly obtain MLLM performance metrics by quizzing fewer question. First, First, we constructed the interview dataset, which was built on an existing MLLM assessment dataset, by adding difficulty labels based on the performance of some typical MLLMs in this dataset. Second, we propose an MLLM Interview strategy, which obtains an initial performance situation of the large model by quizzing a small number of topics and then continuously tries to test the model's limits. Through extensive experiments, the result shows that the MITV strategy proposed in this paper performs well on MLLM benchmark datasets, and it is able to obtain the model evaluation capability faster through a small number of questions and answers.