Private Deep Learning with Teacher Ensembles
This addresses privacy concerns in deep learning for sensitive data, offering a method to reduce performance degradation while ensuring privacy, though it appears incremental as it builds on existing ensemble and distillation techniques.
The paper tackles privacy-preserving deep learning by proposing a teacher ensemble method using differential private stochastic gradient descent with knowledge distillation and hint learning, achieving improved performance and privacy-budget on image datasets like CIFAR10, MINST, and SVHN compared to previous state-of-the-art methods.
Privacy-preserving deep learning is crucial for deploying deep neural network based solutions, especially when the model works on data that contains sensitive information. Most privacy-preserving methods lead to undesirable performance degradation. Ensemble learning is an effective way to improve model performance. In this work, we propose a new method for teacher ensembles that uses more informative network outputs under differential private stochastic gradient descent and provide provable privacy guarantees. Out method employs knowledge distillation and hint learning on intermediate representations to facilitate the training of student model. Additionally, we propose a simple weighted ensemble scheme that works more robustly across different teaching settings. Experimental results on three common image datasets benchmark (i.e., CIFAR10, MINST, and SVHN) demonstrate that our approach outperforms previous state-of-the-art methods on both performance and privacy-budget.