On Optimal Hyperparameters for Differentially Private Deep Transfer Learning
This work addresses a critical issue for practitioners in privacy-preserving machine learning, as it identifies suboptimal hyperparameter choices that can degrade model performance, though it is incremental in refining existing methods.
The paper tackles the problem of selecting optimal hyperparameters for differentially private deep transfer learning, revealing a mismatch between theory and practice for clipping bound and showing that existing batch size heuristics fail, with performance drops of up to 15% in accuracy under tight privacy constraints.
Differentially private (DP) transfer learning, i.e., fine-tuning a pretrained model on private data, is the current state-of-the-art approach for training large models under privacy constraints. We focus on two key hyperparameters in this setting: the clipping bound $C$ and batch size $B$. We show a clear mismatch between the current theoretical understanding of how to choose an optimal $C$ (stronger privacy requires smaller $C$) and empirical outcomes (larger $C$ performs better under strong privacy), caused by changes in the gradient distributions. Assuming a limited compute budget (fixed epochs), we demonstrate that the existing heuristics for tuning $B$ do not work, while cumulative DP noise better explains whether smaller or larger batches perform better. We also highlight how the common practice of using a single $(C,B)$ setting across tasks can lead to suboptimal performance. We find that performance drops especially when moving between loose and tight privacy and between plentiful and limited compute, which we explain by analyzing clipping as a form of gradient re-weighting and examining cumulative DP noise.