Exploring and Bridging Knowledge Holes in Unlearned Multimodal Large Language Models
For researchers and practitioners in MLLM safety, this work provides a more fine-grained evaluation paradigm and a method to mitigate unintended degradation, addressing a critical gap in trustworthy unlearning.
The paper identifies a blind spot in evaluating machine unlearning for Multimodal Large Language Models (MLLMs), where current benchmarks miss 'knowledge holes'—severe degradation on benign inputs similar to the forget set. They propose a benchmark to detect these holes and a method (SPAR) that recovers over 98% of vanilla response quality (vs. below 50% for baselines) while achieving 0.00% attack success rate and competitive utility.
Machine unlearning offers a promising approach to remove unsafe content from Multimodal Large Language Models (MLLMs), yet ensuring the precision of unlearning remains a persistent challenge. One reason is that current MLLM unlearning evaluation paradigms suffer from a critical blind spot: they assess model utility through benchmarks whose representations are distant from the forget set, failing to capture knowledge holes---severe degradation on benign adjacent inputs. To probe knowledge holes in unlearned MLLMs, we construct a benchmark that captures unintended degradation on benign inputs sharing generic patterns with the forget set, and confirm through controlled experiments that they are a systematic consequence of commonly used approaches. Furthermore, to bridge this gap, we propose Selective Protection with Anchored Regularization, which protects generic patterns via anchored activation filtering while reinforcing them through entity-abstracted enhancement. Our experiments on SafeEraser demonstrate that SPAR recovers over 98% of vanilla response quality compared to below 50% for standard baselines---while achieving 0.00% attack success rate and competitive model utility. These results underscore the necessity of more fine-grained evaluation for trustworthy MLLM unlearning.