What can we learn from Data Leakage and Unlearning for Law?
This work highlights privacy and legal concerns for companies using LLMs, identifying incremental risks in data leakage and unlearning processes.
The paper tackles the problem of data leakage in large language models (LLMs), showing that fine-tuned models leak both fine-tuning and pre-training data, including personally identifiable information (PII), and that unlearning vulnerable data points exposes new ones to extraction. This poses privacy and legal risks for companies using LLMs.
Large Language Models (LLMs) have a privacy concern because they memorize training data (including personally identifiable information (PII) like emails and phone numbers) and leak it during inference. A company can train an LLM on its domain-customized data which can potentially also include their users' PII. In order to comply with privacy laws such as the "right to be forgotten", the data points of users that are most vulnerable to extraction could be deleted. We find that once the most vulnerable points are deleted, a new set of points become vulnerable to extraction. So far, little attention has been given to understanding memorization for fine-tuned models. In this work, we also show that not only do fine-tuned models leak their training data but they also leak the pre-training data (and PII) memorized during the pre-training phase. The property of new data points becoming vulnerable to extraction after unlearning and leakage of pre-training data through fine-tuned models can pose significant privacy and legal concerns for companies that use LLMs to offer services. We hope this work will start an interdisciplinary discussion within AI and law communities regarding the need for policies to tackle these issues.