Data Efficient Subset Training with Differential Privacy
This work addresses the challenge of efficient model training under differential privacy constraints, but it is incremental as it adapts an existing method and reports negative results.
The paper tackled the problem of slow convergence and extensive hyperparameter tuning in differentially private machine learning by adapting the GLISTER data-efficient subset training method to the private setting, but found that practical privacy budgets are too restrictive for effective data-efficient training.
Private machine learning introduces a trade-off between the privacy budget and training performance. Training convergence is substantially slower and extensive hyper parameter tuning is required. Consequently, efficient methods to conduct private training of models is thoroughly investigated in the literature. To this end, we investigate the strength of the data efficient model training methods in the private training setting. We adapt GLISTER (Killamsetty et al., 2021b) to the private setting and extensively assess its performance. We empirically find that practical choices of privacy budgets are too restrictive for data efficient training in the private setting.