Machine Unlearning via Information Theoretic Regularization
This work addresses the problem of machine unlearning for machine learning and AI practitioners, providing a solution that minimizes utility loss and ensures rigorous guarantees.
This paper tackles the problem of removing undesirable information from a learning outcome, achieving a unified solution for feature unlearning and providing provable guarantees for data point unlearning. The result is a highly adaptable framework for machine learning and AI applications.
How can we effectively remove or "unlearn" undesirable information, such as specific features or individual data points, from a learning outcome while minimizing utility loss and ensuring rigorous guarantees? We introduce a mathematical framework based on information-theoretic regularization to address both feature and data point unlearning. For feature unlearning, we derive a unified solution that simultaneously optimizes diverse learning objectives, including entropy, conditional entropy, KL-divergence, and the energy of conditional probability. For data point unlearning, we first propose a novel definition that serves as a practical condition for unlearning via retraining, is easy to verify, and aligns with the principles of differential privacy from an inference perspective. Then, we provide provable guarantees for our framework on data point unlearning. By combining flexibility in learning objectives with simplicity in regularization design, our approach is highly adaptable and practical for a wide range of machine learning and AI applications.