On the utility and protection of optimization with differential privacy and classic regularization techniques
This work addresses privacy preservation in deep learning for model developers, but it is incremental as it compares existing methods rather than introducing new ones.
The paper compares differentially-private stochastic gradient descent (DP-SGD) with standard regularization techniques like dropout and l2-regularization, finding that the latter often provide superior privacy protection against attacks such as membership inference and model inversion, while DP-SGD can degrade model performance.
Nowadays, owners and developers of deep learning models must consider stringent privacy-preservation rules of their training data, usually crowd-sourced and retaining sensitive information. The most widely adopted method to enforce privacy guarantees of a deep learning model nowadays relies on optimization techniques enforcing differential privacy. According to the literature, this approach has proven to be a successful defence against several models' privacy attacks, but its downside is a substantial degradation of the models' performance. In this work, we compare the effectiveness of the differentially-private stochastic gradient descent (DP-SGD) algorithm against standard optimization practices with regularization techniques. We analyze the resulting models' utility, training performance, and the effectiveness of membership inference and model inversion attacks against the learned models. Finally, we discuss differential privacy's flaws and limits and empirically demonstrate the often superior privacy-preserving properties of dropout and l2-regularization.