ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing
This work addresses the need for a more robust and unbiased evaluation framework for multimodal knowledge editing, which is crucial for researchers and developers working on improving the accuracy and reliability of MLLMs, though it is incremental in nature.
The authors tackled the problem of outdated or inaccurate information in large multimodal language models by introducing ComprehendEdit, a comprehensive benchmark with eight diverse tasks and two novel metrics (KGI and KPI) to evaluate editing effects, and established a baseline method (HICE) that demonstrates improved performance.
Large multimodal language models (MLLMs) have revolutionized natural language processing and visual understanding, but often contain outdated or inaccurate information. Current multimodal knowledge editing evaluations are limited in scope and potentially biased, focusing on narrow tasks and failing to assess the impact on in-domain samples. To address these issues, we introduce ComprehendEdit, a comprehensive benchmark comprising eight diverse tasks from multiple datasets. We propose two novel metrics: Knowledge Generalization Index (KGI) and Knowledge Preservation Index (KPI), which evaluate editing effects on in-domain samples without relying on AI-synthetic samples. Based on insights from our framework, we establish Hierarchical In-Context Editing (HICE), a baseline method employing a two-stage approach that balances performance across all metrics. This study provides a more comprehensive evaluation framework for multimodal knowledge editing, reveals unique challenges in this field, and offers a baseline method demonstrating improved performance. Our work opens new perspectives for future research and provides a foundation for developing more robust and effective editing techniques for MLLMs. The ComprehendEdit benchmark and implementation code are available at https://github.com/yaohui120/ComprehendEdit.