A Reliable Cryptographic Framework for Empirical Machine Unlearning Evaluation
This work addresses the need for trustworthy evaluation methods in machine unlearning, crucial for compliance with data protection regulations, though it is incremental as it builds on existing MIA-based approaches.
The paper tackles the problem of reliably evaluating machine unlearning algorithms, which remove specific training data, by addressing pitfalls in existing membership inference attack-based metrics. It proposes a cryptographic game framework that provides provable guarantees and demonstrates effectiveness through theoretical and empirical analysis.
Machine unlearning updates machine learning models to remove information from specific training samples, complying with data protection regulations that allow individuals to request the removal of their personal data. Despite the recent development of numerous unlearning algorithms, reliable evaluation of these algorithms remains an open research question. In this work, we focus on membership inference attack (MIA) based evaluation, one of the most common approaches for evaluating unlearning algorithms, and address various pitfalls of existing evaluation metrics lacking theoretical understanding and reliability. Specifically, by modeling the proposed evaluation process as a \emph{cryptographic game} between unlearning algorithms and MIA adversaries, the naturally induced evaluation metric measures the data removal efficacy of unlearning algorithms and enjoys provable guarantees that existing evaluation metrics fail to satisfy. Furthermore, we propose a practical and efficient approximation of the induced evaluation metric and demonstrate its effectiveness through both theoretical analysis and empirical experiments. Overall, this work presents a novel and reliable approach to empirically evaluating unlearning algorithms, paving the way for the development of more effective unlearning techniques.