DP-SGD with weight clipping
This work addresses privacy concerns in deep learning training for applications requiring data protection, though it appears incremental as it builds on existing DP-SGD methods.
The paper tackles the problem of bias in differentially private gradient descent methods by introducing a novel approach that uses public Lipschitz bounds to refine noise adjustments, resulting in improved differential privacy guarantees and empirical performance over existing methods.
Recently, due to the popularity of deep neural networks and other methods whose training typically relies on the optimization of an objective function, and due to concerns for data privacy, there is a lot of interest in differentially private gradient descent methods. To achieve differential privacy guarantees with a minimum amount of noise, it is important to be able to bound precisely the sensitivity of the information which the participants will observe. In this study, we present a novel approach that mitigates the bias arising from traditional gradient clipping. By leveraging a public upper bound of the Lipschitz value of the current model and its current location within the search domain, we can achieve refined noise level adjustments. We present a new algorithm with improved differential privacy guarantees and a systematic empirical evaluation, showing that our new approach outperforms existing approaches also in practice.