Additive Logistic Mechanism for Privacy-Preserving Self-Supervised Learning
This addresses privacy protection for machine learning models using sensitive data, but is incremental as it builds on existing differential privacy mechanisms.
The paper tackled privacy risks in self-supervised learning by showing that fine-tuning is vulnerable to attacks, and proposed a novel differential privacy mechanism using logistic-distributed noise that reduces attack accuracy to near 50% while keeping performance loss under 5%.
We study the privacy risks that are associated with training a neural network's weights with self-supervised learning algorithms. Through empirical evidence, we show that the fine-tuning stage, in which the network weights are updated with an informative and often private dataset, is vulnerable to privacy attacks. To address the vulnerabilities, we design a post-training privacy-protection algorithm that adds noise to the fine-tuned weights and propose a novel differential privacy mechanism that samples noise from the logistic distribution. Compared to the two conventional additive noise mechanisms, namely the Laplace and the Gaussian mechanisms, the proposed mechanism uses a bell-shaped distribution that resembles the distribution of the Gaussian mechanism, and it satisfies pure $ε$-differential privacy similar to the Laplace mechanism. We apply membership inference attacks on both unprotected and protected models to quantify the trade-off between the models' privacy and performance. We show that the proposed protection algorithm can effectively reduce the attack accuracy to roughly 50\%-equivalent to random guessing-while maintaining a performance loss below 5\%.