Randomized Masked Finetuning: An Efficient Way to Mitigate Memorization of PIIs in LLMs
This addresses privacy risks for users of LLMs by mitigating PII memorization, though it appears incremental as a fine-tuning technique building on existing privacy-preserving methods.
The paper tackles the problem of LLMs memorizing personally identifying information (PIIs) from training data, which poses security and privacy risks, by introducing Randomized Masked Fine-Tuning (RMFT) that reduces PII extraction rates by over 80% while maintaining minimal performance impact with only a 5.73% increase in perplexity.
The current literature on memorization in Natural Language Models, especially Large Language Models (LLMs), poses severe security and privacy risks, as models tend to memorize personally identifying information (PIIs) from training data. We introduce Randomized Masked Fine-Tuning (RMFT), a novel privacy-preserving fine-tuning technique that reduces PII memorization while minimizing performance impact. Using the Enron Email Dataset, we demonstrate that RMFT achieves an 80.81% reduction in Total Extraction Rate and 80.17% reduction in Seen Extraction Rate compared to baseline fine-tuning, outperforming deduplication methods while maintaining only a 5.73% increase in perplexity. We present MaxTER, a Pareto-optimal evaluation framework for assessing privacy-utility tradeoffs, and show the performance of RMFT vs Deduplication by Area Under The Response Curve (AURC) metric.