MedMKEB: A Comprehensive Knowledge Editing Benchmark for Medical Multimodal Large Language Models
This addresses the problem of keeping medical AI models up-to-date without retraining for researchers and practitioners, but it is incremental as it focuses on benchmarking rather than a new editing method.
The authors tackled the lack of systematic benchmarks for editing outdated or incorrect knowledge in medical multimodal large language models (MLLMs) by introducing MedMKEB, a comprehensive benchmark built on a medical visual question-answering dataset, which revealed limitations in existing editing approaches through extensive experiments.
Recent advances in multimodal large language models (MLLMs) have significantly improved medical AI, enabling it to unify the understanding of visual and textual information. However, as medical knowledge continues to evolve, it is critical to allow these models to efficiently update outdated or incorrect information without retraining from scratch. Although textual knowledge editing has been widely studied, there is still a lack of systematic benchmarks for multimodal medical knowledge editing involving image and text modalities. To fill this gap, we present MedMKEB, the first comprehensive benchmark designed to evaluate the reliability, generality, locality, portability, and robustness of knowledge editing in medical multimodal large language models. MedMKEB is built on a high-quality medical visual question-answering dataset and enriched with carefully constructed editing tasks, including counterfactual correction, semantic generalization, knowledge transfer, and adversarial robustness. We incorporate human expert validation to ensure the accuracy and reliability of the benchmark. Extensive single editing and sequential editing experiments on state-of-the-art general and medical MLLMs demonstrate the limitations of existing knowledge-based editing approaches in medicine, highlighting the need to develop specialized editing strategies. MedMKEB will serve as a standard benchmark to promote the development of trustworthy and efficient medical knowledge editing algorithms.