Fast Exact Unlearning for In-Context Learning Data for LLMs
This addresses the challenge of retroactive data removal for LLMs, which is crucial for privacy and compliance, though it is incremental as it builds on existing unlearning concepts.
The paper tackles the problem of efficiently removing specific training data from large language models by introducing a method for exact unlearning of fine-tuning data using in-context learning with quantized k-means, achieving similar performance to fine-tuning but with vastly reduced unlearning costs.
Modern machine learning models are expensive to train, and there is a growing concern about the challenge of retroactively removing specific training data. Achieving exact unlearning in deep learning pipelines--producing models as if certain data had never been included in training--remains an open problem. In this paper, we revisit exact unlearning in deep learning and show that for large language models (LLMs) we can efficiently exactly unlearn "fine-tuning data" (the data used to adapt a pre-trained model). This follows from two observations. First, we can use in-context learning to adapt the LLM to the fine-tuning dataset instead of SGD based algorithms. Second, we show that accurate in-context learning can be done with quantized k-means, which allows for effectively constant time unlearning operations. Our evaluation shows that this unlearning recipe has similar performance to fine-tuning alternatives, but vastly reduces the unlearning costs. Our study also highlights the need for new measures of unlearning cost when adapting the learning algorithm to have faster unlearn operations.