Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearning
This addresses privacy concerns in MLLMs for applications like image-based question answering, but it is incremental as it builds on existing unlearning methods by incorporating multimodal cues.
The paper tackles the problem of preventing Multimodal Large Language Models (MLLMs) from revealing private information when queried about target images by proposing Visual-Guided Key-Token Regularization (ViKeR), which leverages visual cues to prioritize key tokens during unlearning, resulting in effective unlearning while mitigating forgetting and maintaining response coherence on MLLMU and CLEAR benchmarks.
Unlearning in Multimodal Large Language Models (MLLMs) prevents the model from revealing private information when queried about target images. Existing MLLM unlearning methods largely adopt approaches developed for LLMs. They treat all answer tokens uniformly, disregarding their varying importance in the unlearning process. Moreover, these methods focus exclusively on the language modality, disregarding visual cues that indicate key tokens in answers. In this paper, after formulating the problem of unlearning in multimodal question answering for MLLMs, we propose Visual-Guided Key-Token Regularization (ViKeR). We leverage irrelevant visual inputs to predict ideal post-unlearning token-level distributions and use these distributions to regularize the unlearning process, thereby prioritizing key tokens. Further, we define key tokens in unlearning via information entropy and discuss ViKeR's effectiveness through token-level gradient reweighting, which amplifies updates on key tokens. Experiments on MLLMU and CLEAR benchmarks demonstrate that our method effectively performs unlearning while mitigating forgetting and maintaining response coherence.