Mitigating Noise Detriment in Differentially Private Federated Learning with Model Pre-training
This work addresses the privacy-utility trade-off in federated learning for applications requiring data privacy, though it is incremental as it builds on existing pre-training approaches.
The paper tackled the problem of reduced accuracy in Differentially Private Federated Learning (DPFL) due to noise perturbation by proposing Pretrain-DPFL, a framework that systematically evaluates fine-tuning strategies for pre-trained models, resulting in up to 25.22% higher accuracy than scratch training and outperforming baselines by 8.19%.
Differentially Private Federated Learning (DPFL) strengthens privacy protection by perturbing model gradients with noise, though at the cost of reduced accuracy. Although prior empirical studies indicate that initializing from pre-trained rather than random parameters can alleviate noise disturbance, the problem of optimally fine-tuning pre-trained models in DPFL remains unaddressed. In this paper, we propose Pretrain-DPFL, a framework that systematically evaluates three most representative fine-tuning strategies: full-tuning (FT), head-tuning (HT), and unified-tuning(UT) combining HT followed by FT. Through convergence analysis under smooth non-convex loss, we establish theoretical conditions for identifying the optimal fine-tuning strategy in Pretrain-DPFL, thereby maximizing the benefits of pre-trained models in mitigating noise disturbance. Extensive experiments across multiple datasets demonstrate Pretrain-DPFL's superiority, achieving $25.22\%$ higher accuracy than scratch training and outperforming the second-best baseline by $8.19\%$, significantly improving the privacy-utility trade-off in DPFL.