Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation
For researchers and practitioners working on privacy in multilingual LLMs, this work addresses a gap in evaluation by providing metrics that capture cross-linguistic information removal, though it is an incremental improvement over existing per-language protocols.
The paper identifies limitations in existing evaluations of Multilingual Machine Unlearning (MMU) for LLMs, which fail to capture cross-linguistic information spread. It proposes two new metrics, Knowledge Separability Score (KSS) and Knowledge Persistence Score (KPS), to evaluate unlearning quality across languages, and demonstrates their utility through comprehensive analyses.
While LLMs are increasingly used in commercial services, they pose privacy risks such as leakage of sensitive personally identifiable information (PII). For LLMs trained on multilingual corpora, Multilingual Machine Unlearning (MMU) aims to remove information across multiple languages. However, prior MMU evaluations fail to capture such cross-linguistic distribution of information, being largely limited to direct extensions of per-language evaluation protocols. To this end, we propose two metrics to evaluate the information spread across languages: the Knowledge Separability Score (KSS) and the Knowledge Persistence Score (KPS). KSS measures the overall unlearning quality across multiple languages, while KPS more specifically aims to assess consistent removal of information among different language pairs. We evaluated various unlearning methods in the multilingual setting with these metrics and conducted comprehensive analyses. Through our investigation, we provide insights into unique phenomena exclusive to MMU and offer a new perspective on MMU evaluation.