Empirical Calibration and Metric Differential Privacy in Language Models
This work addresses the challenge of comparing privacy guarantees across frameworks in NLP, which is important for researchers and practitioners in privacy-preserving machine learning, though it is incremental as it adapts calibration methods from image processing to NLP.
The paper tackled the problem of calibrating differential privacy guarantees in NLP models by showing that reconstruction attacks are more effective than membership inference attacks for empirical calibration. As a result, they introduced a novel directional privacy mechanism based on the von Mises-Fisher distribution, demonstrating that different mechanisms have varying strengths in utility-privacy trade-offs.
NLP models trained with differential privacy (DP) usually adopt the DP-SGD framework, and privacy guarantees are often reported in terms of the privacy budget $ε$. However, $ε$ does not have any intrinsic meaning, and it is generally not possible to compare across variants of the framework. Work in image processing has therefore explored how to empirically calibrate noise across frameworks using Membership Inference Attacks (MIAs). However, this kind of calibration has not been established for NLP. In this paper, we show that MIAs offer little help in calibrating privacy, whereas reconstruction attacks are more useful. As a use case, we define a novel kind of directional privacy based on the von Mises-Fisher (VMF) distribution, a metric DP mechanism that perturbs angular distance rather than adding (isotropic) Gaussian noise, and apply this to NLP architectures. We show that, even though formal guarantees are incomparable, empirical privacy calibration reveals that each mechanism has different areas of strength with respect to utility-privacy trade-offs.