Provable unlearning in topic modeling and downstream tasks
This addresses legal concerns around data provenance for researchers and practitioners using topic models, though it is incremental as it extends existing unlearning guarantees to a new setting.
The paper tackles the problem of verifying machine unlearning in pre-training and fine-tuning by providing the first theoretical guarantees for topic models, designing an algorithm with computational overhead independent of dataset size and showing it is easier to unlearn pre-training data after fine-tuning without modifying the base model.
Machine unlearning algorithms are increasingly important as legal concerns arise around the provenance of training data, but verifying the success of unlearning is often difficult. Provable guarantees for unlearning are often limited to supervised learning settings. In this paper, we provide the first theoretical guarantees for unlearning in the pre-training and fine-tuning paradigm by studying topic models, simple bag-of-words language models that can be adapted to solve downstream tasks like retrieval and classification. First, we design a provably effective unlearning algorithm for topic models that incurs a computational overhead independent of the size of the original dataset. Our analysis additionally quantifies the deletion capacity of the model -- i.e., the number of examples that can be unlearned without incurring a significant cost in model performance. Finally, we formally extend our analyses to account for adaptation to a given downstream task. In particular, we design an efficient algorithm to perform unlearning after fine-tuning the topic model via a linear head. Notably, we show that it is easier to unlearn pre-training data from models that have been fine-tuned to a particular task, and one can unlearn this data without modifying the base model.