From Teacher to Student: Tracking Memorization Through Model Distillation
This addresses privacy and security concerns for users of LLMs by showing a method to mitigate memorization, though it appears incremental as it builds on existing distillation techniques.
The study tackled the problem of memorization in large language models by investigating how knowledge distillation from a fine-tuned teacher to a smaller student affects memorization risks, finding that it significantly reduces these risks compared to standard fine-tuning.
Large language models (LLMs) are known to memorize parts of their training data, raising important concerns around privacy and security. While previous research has focused on studying memorization in pre-trained models, much less is known about how knowledge distillation (KD) affects memorization.In this study, we explore how different KD methods influence the memorization of fine-tuned task data when a large teacher model is distilled into smaller student variants.This study demonstrates that distilling a larger teacher model, fine-tuned on a dataset, into a smaller variant not only lowers computational costs and model size but also significantly reduces the memorization risks compared to standard fine-tuning approaches.