Hierarchical Federated Unlearning for Large Language Models
This addresses privacy and security concerns for LLM applications by enabling scalable, privacy-preserving unlearning, though it is incremental as it builds on existing federated and unlearning methods.
The paper tackled the problem of removing undesirable knowledge from large language models (LLMs) in decentralized settings with continuous, heterogeneous unlearning needs, proposing a federated unlearning approach that effectively handles such requests while maintaining strong model utility, as shown in experiments on benchmarks like WMDP, MUSE, and TOFU.
Large Language Models (LLMs) are increasingly integrated into real-world applications, raising concerns about privacy, security and the need to remove undesirable knowledge. Machine Unlearning has emerged as a promising solution, yet faces two key challenges: (1) practical unlearning needs are often continuous and heterogeneous, and (2) they involve decentralized, sensitive data with asymmetric access. These factors result in inter-domain and intra-domain interference, which further amplifies the dilemma of unbalanced forgetting and retaining performance. In response, we propose a federated unlearning approach for LLMs that is scalable and privacy preserving. Our method decouples unlearning and retention via task-specific adapter learning and employs a hierarchical merging strategy to mitigate conflicting objectives and enables robust, adaptable unlearning updates. Comprehensive experiments on benchmarks of WMDP, MUSE, and TOFU showed that our approach effectively handles heterogeneous unlearning requests while maintaining strong LLM utility compared with baseline methods.