Forgetting Private Textual Sequences in Language Models via Leave-One-Out Ensemble
This addresses the practical need for data deletion rights in deployed models, offering a computationally efficient alternative to full retraining, though it is incremental as it builds on existing teacher-student frameworks.
The paper tackles the problem of efficiently removing memorized personal information from language models upon user requests, proposing a leave-one-out ensemble method that achieves superior privacy-utility trade-offs compared to other methods, as demonstrated on LibriSpeech and WikiText-103 datasets.
Recent research has shown that language models have a tendency to memorize rare or unique token sequences in the training corpus. After deploying a model, practitioners might be asked to delete any personal information from the model by individuals' requests. Re-training the underlying model every time individuals would like to practice their rights to be forgotten is computationally expensive. We employ a teacher-student framework and propose a novel leave-one-out ensemble method to unlearn the targeted textual sequences that need to be forgotten from the model. In our approach, multiple teachers are trained on disjoint sets; for each targeted sequence to be removed, we exclude the teacher trained on the set containing this sequence and aggregate the predictions from remaining teachers to provide supervision during fine-tuning. Experiments on LibriSpeech and WikiText-103 datasets show that the proposed method achieves superior privacy-utility trade-offs than other counterparts.