MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs
This addresses the problem of evaluating instruction fidelity in multimodal LLMs for researchers and developers, though it is incremental as it builds on existing benchmark methodologies.
The authors introduced MIA-Bench, a benchmark with 400 image-prompt pairs to evaluate multimodal LLMs on strict instruction following, revealing significant performance variations among state-of-the-art models.
We introduce MIA-Bench, a new benchmark designed to evaluate multimodal large language models (MLLMs) on their ability to strictly adhere to complex instructions. Our benchmark comprises a diverse set of 400 image-prompt pairs, each crafted to challenge the models' compliance with layered instructions in generating accurate responses that satisfy specific requested patterns. Evaluation results from a wide array of state-of-the-art MLLMs reveal significant variations in performance, highlighting areas for improvement in instruction fidelity. Additionally, we create extra training data and explore supervised fine-tuning to enhance the models' ability to strictly follow instructions without compromising performance on other tasks. We hope this benchmark not only serves as a tool for measuring MLLM adherence to instructions, but also guides future developments in MLLM training methods.