One size does not fit all: Investigating strategies for differentially-private learning across NLP tasks
This work addresses the challenge of balancing privacy and performance in NLP for sensitive data, but it is incremental as it builds on existing DP-SGD research.
The paper investigates differentially-private learning strategies across NLP tasks, finding that unlike non-private approaches where larger models perform better, privacy-preserving methods require task-specific treatments to achieve adequate performance, with no consistent winning pattern.
Preserving privacy in contemporary NLP models allows us to work with sensitive data, but unfortunately comes at a price. We know that stricter privacy guarantees in differentially-private stochastic gradient descent (DP-SGD) generally degrade model performance. However, previous research on the efficiency of DP-SGD in NLP is inconclusive or even counter-intuitive. In this short paper, we provide an extensive analysis of different privacy preserving strategies on seven downstream datasets in five different `typical' NLP tasks with varying complexity using modern neural models based on BERT and XtremeDistil architectures. We show that unlike standard non-private approaches to solving NLP tasks, where bigger is usually better, privacy-preserving strategies do not exhibit a winning pattern, and each task and privacy regime requires a special treatment to achieve adequate performance.