LUME: LLM Unlearning with Multitask Evaluations
This work addresses the need for effective unlearning methods in LLMs to handle privacy and copyright issues, but it is incremental as it focuses on benchmarking rather than introducing a new unlearning method.
The paper tackles the problem of removing copyrighted, sensitive, or private content from large language models without full retraining by developing a multi-task unlearning benchmark called LUME, which includes three tasks and evaluates several unlearning algorithms on fine-tuned LLMs of 1B and 7B parameters.
Unlearning aims to remove copyrighted, sensitive, or private content from large language models (LLMs) without a full retraining. In this work, we develop a multi-task unlearning benchmark (LUME) which features three tasks: (1) unlearn synthetically generated creative short novels, (2) unlearn synthetic biographies with sensitive information, and (3) unlearn a collection of public biographies. We further release two fine-tuned LLMs of 1B and 7B parameter sizes as the target models. We conduct detailed evaluations of several recently proposed unlearning algorithms and present results on carefully crafted metrics to understand their behavior and limitations.