Module-Aware Parameter-Efficient Machine Unlearning on Transformers
This work addresses the need for efficient data removal to comply with privacy regulations in Transformer-based models, representing an incremental improvement over existing methods.
The paper tackles the problem of inaccurate parameter identification in parameter-efficient machine unlearning for Transformers, proposing MAPE-Unlearn, which uses learnable masks to pinpoint influence-critical parameters and achieves effective unlearning performance in experiments.
Transformer has become fundamental to a vast series of pre-trained large models that have achieved remarkable success across diverse applications. Machine unlearning, which focuses on efficiently removing specific data influences to comply with privacy regulations, shows promise in restricting updates to influence-critical parameters. However, existing parameter-efficient unlearning methods are largely devised in a module-oblivious manner, which tends to inaccurately identify these parameters and leads to inferior unlearning performance for Transformers. In this paper, we propose {\tt MAPE-Unlearn}, a module-aware parameter-efficient machine unlearning approach that uses a learnable pair of masks to pinpoint influence-critical parameters in the heads and filters of Transformers. The learning objective of these masks is derived by desiderata of unlearning and optimized through an efficient algorithm featured by a greedy search with a warm start. Extensive experiments on various Transformer models and datasets demonstrate the effectiveness and robustness of {\tt MAPE-Unlearn} for unlearning.