Differentially Private Model Publishing for Deep Learning
This addresses privacy risks for individuals when sharing pre-trained models, but it is incremental as it builds on existing differential privacy methods.
The paper tackles the problem of privacy leakage in deep learning models trained on sensitive data by proposing a differentially private approach that uses concentrated differential privacy and dynamic privacy budget allocation, achieving improved privacy loss accounting, training efficiency, and model quality under a given privacy budget.
Deep learning techniques based on neural networks have shown significant success in a wide range of AI tasks. Large-scale training datasets are one of the critical factors for their success. However, when the training datasets are crowdsourced from individuals and contain sensitive information, the model parameters may encode private information and bear the risks of privacy leakage. The recent growing trend of the sharing and publishing of pre-trained models further aggravates such privacy risks. To tackle this problem, we propose a differentially private approach for training neural networks. Our approach includes several new techniques for optimizing both privacy loss and model accuracy. We employ a generalization of differential privacy called concentrated differential privacy(CDP), with both a formal and refined privacy loss analysis on two different data batching methods. We implement a dynamic privacy budget allocator over the course of training to improve model accuracy. Extensive experiments demonstrate that our approach effectively improves privacy loss accounting, training efficiency and model quality under a given privacy budget.