Differentially Private Model Merging
For practitioners needing to adapt privacy levels post-hoc without retraining, this provides a flexible solution, though the approach is incremental.
This work proposes post-processing techniques (random selection and linear combination) to generate models satisfying any target differential privacy requirement from a set of pre-trained models with different privacy/utility tradeoffs, without additional training. The linear combination method is shown to be superior both theoretically and empirically.
In machine learning applications, privacy requirements during inference or deployment time could change constantly due to varying policies, regulations, or user experience. In this work, we aim to generate a magnitude of models to satisfy any target differential privacy (DP) requirement without additional training steps, given a set of existing models trained on the same dataset with different privacy/utility tradeoffs. We propose two post processing techniques, namely random selection and linear combination, to output a final private model for any target privacy parameter. We provide privacy accounting of these approaches from the lens of R'enyi DP and privacy loss distributions for general problems. In a case study on private mean estimation, we fully characterize the privacy/utility results and theoretically establish the superiority of linear combination over random selection. Empirically, we validate our approach and analyses on several models and both synthetic and real-world datasets.