On the Importance of Difficulty Calibration in Membership Inference Attacks
This work addresses the unreliability of membership inference attacks for privacy assessment in machine learning, though it is incremental as it builds on existing attacks.
The paper tackled the problem of high false positive rates in membership inference attacks by proposing difficulty calibration, which adjusts membership scores based on sample difficulty, and showed it significantly reduces false positive rates without accuracy loss.
The vulnerability of machine learning models to membership inference attacks has received much attention in recent years. However, existing attacks mostly remain impractical due to having high false positive rates, where non-member samples are often erroneously predicted as members. This type of error makes the predicted membership signal unreliable, especially since most samples are non-members in real world applications. In this work, we argue that membership inference attacks can benefit drastically from \emph{difficulty calibration}, where an attack's predicted membership score is adjusted to the difficulty of correctly classifying the target sample. We show that difficulty calibration can significantly reduce the false positive rate of a variety of existing attacks without a loss in accuracy.