DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models
Provides a practical solution for improving accuracy in differentially private federated learning of LLMs, addressing the bottleneck of hyperparameter tuning in adaptive clipping.
DP-LAC introduces a lightweight adaptive clipping method for differentially private federated fine-tuning of language models, achieving an average accuracy gain of 6.6% over state-of-the-art methods without additional privacy budget or hyperparameter tuning.
Federated learning (FL) enables the collaborative training of large-scale language models (LLMs) across edge devices while keeping user data on-device. However, FL still exposes sensitive information through client-provided gradients. Differentially private stochastic gradient descent (DP-SGD) mitigates this risk by clipping each client's contribution to a threshold $C$ and adding noise proportional to $C$. Existing adaptive clipping techniques dynamically adjust $C$ but demand tedious hyperparameter tuning, which can erode the privacy budget. In this paper, we introduce DP-LAC, a method that first estimates an initial clipping threshold within an order of magnitude of the optimum using private histogram estimation, and then adapts this threshold during training without consuming additional privacy budget or introducing new hyperparameters. Empirical results show that DP-LAC outperforms both state-of-the-art adaptive clipping methods and vanilla DP-SGD, achieving an average accuracy gain of $6.6\%$.