DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models
This addresses privacy concerns for users in on-device LLM systems, but it is incremental as it builds on existing federated learning and LoRA methods.
The paper tackled the problem of privacy risks in federated fine-tuning for on-device large language models by proposing DP-FedLoRA, which integrates LoRA-based adaptation with differential privacy, achieving competitive performance on benchmarks while providing strong privacy guarantees.
As on-device large language model (LLM) systems become increasingly prevalent, federated fine-tuning enables advanced language understanding and generation directly on edge devices; however, it also involves processing sensitive, user-specific data, raising significant privacy concerns within the federated learning framework. To address these challenges, we propose DP-FedLoRA, a privacy-enhanced federated fine-tuning framework that integrates LoRA-based adaptation with differential privacy in a communication-efficient setting. Each client locally clips and perturbs its LoRA matrices using Gaussian noise to satisfy ($ε$, $δ$)-differential privacy. We further provide a theoretical analysis demonstrating the unbiased nature of the updates and deriving bounds on the variance introduced by noise, offering practical guidance for privacy-budget calibration. Experimental results across mainstream benchmarks show that DP-FedLoRA delivers competitive performance while offering strong privacy guarantees, paving the way for scalable and privacy-preserving LLM deployment in on-device environments.