Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping
This addresses evaluation validity issues for researchers and developers working with LVLMs, though it is incremental as it builds on existing benchmarks.
The paper tackles the problem of static multimodal evaluation benchmarks for Large Vision-Language Models (LVLMs) by introducing Vision-Language Bootstrapping (VLB), a dynamic protocol that reduces data contamination and exposes performance limitations across benchmarks like SEEDBench, MMBench, and MME.
Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across multimodal tasks such as visual perception and reasoning, leading to good performance on various multimodal evaluation benchmarks. However, these benchmarks keep a static nature and overlap with the pre-training data, resulting in fixed complexity constraints and data contamination issues. This raises the concern regarding the validity of the evaluation. To address these two challenges, we introduce a dynamic multimodal evaluation protocol called Vision-Language Bootstrapping (VLB). VLB provides a robust and comprehensive assessment for LVLMs with reduced data contamination and flexible complexity. To this end, VLB dynamically generates new visual question-answering samples through a multimodal bootstrapping module that modifies both images and language, while ensuring that newly generated samples remain consistent with the original ones by a judge module. By composing various bootstrapping strategies, VLB offers dynamic variants of existing benchmarks with diverse complexities, enabling the evaluation to co-evolve with the ever-evolving capabilities of LVLMs. Extensive experimental results across multiple benchmarks, including SEEDBench, MMBench, and MME, show that VLB significantly reduces data contamination and exposes performance limitations of LVLMs.