A Closer Look at the Calibration of Differentially Private Learners
This addresses calibration issues in privacy-preserving machine learning, which is crucial for reliable deployment in sensitive domains, though it is incremental as it adapts existing calibration methods to differential privacy.
The paper tackled the problem of miscalibration in classifiers trained with differentially private stochastic gradient descent (DP-SGD) across vision and language tasks, finding that per-example gradient clipping is a major cause and showing that differentially private post-processing methods like temperature scaling and Platt scaling reduce calibration error by an average 3.1-fold with minimal accuracy loss.
We systematically study the calibration of classifiers trained with differentially private stochastic gradient descent (DP-SGD) and observe miscalibration across a wide range of vision and language tasks. Our analysis identifies per-example gradient clipping in DP-SGD as a major cause of miscalibration, and we show that existing approaches for improving calibration with differential privacy only provide marginal improvements in calibration error while occasionally causing large degradations in accuracy. As a solution, we show that differentially private variants of post-processing calibration methods such as temperature scaling and Platt scaling are surprisingly effective and have negligible utility cost to the overall model. Across 7 tasks, temperature scaling and Platt scaling with DP-SGD result in an average 3.1-fold reduction in the in-domain expected calibration error and only incur at most a minor percent drop in accuracy.