MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge
This addresses the need for updating evolving knowledge in multimodal AI models, though it is incremental as it focuses on evaluation rather than a new method.
The paper tackles the problem of multimodal models' knowledge becoming outdated by introducing MMKU-Bench, a benchmark with over 25k knowledge instances and 49k images, and finds that supervised fine-tuning and reinforcement learning from human feedback suffer from catastrophic forgetting, while knowledge editing preserves general capabilities but has limitations in continual updating.
As real-world knowledge continues to evolve, the parametric knowledge acquired by multimodal models during pretraining becomes increasingly difficult to remain consistent with real-world knowledge. Existing research on multimodal knowledge updating focuses only on learning previously unknown knowledge, while overlooking the need to update knowledge that the model has already mastered but that later changes; moreover, evaluation is limited to the same modality, lacking a systematic analysis of cross-modal consistency. To address these issues, this paper proposes MMKU-Bench, a comprehensive evaluation benchmark for multimodal knowledge updating, which contains over 25k knowledge instances and more than 49k images, covering two scenarios, updated knowledge and unknown knowledge, thereby enabling comparative analysis of learning across different knowledge types. On this benchmark, we evaluate a variety of representative approaches, including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and knowledge editing (KE). Experimental results show that SFT and RLHF are prone to catastrophic forgetting, while KE better preserve general capabilities but exhibit clear limitations in continual updating. Overall, MMKU-Bench provides a reliable and comprehensive evaluation benchmark for multimodal knowledge updating, advancing progress in this field.