A Critical Review on the Effectiveness and Privacy Threats of Membership Inference Attacks
This work addresses the issue for ML practitioners and researchers by showing that current reliance on MIAs as a privacy metric can lead to unnecessary model utility sacrifices, indicating an incremental contribution in refining privacy assessment methods.
The paper tackles the problem of overestimating privacy risks from membership inference attacks (MIAs) in machine learning by proposing an evaluation framework to define realistic conditions, finding that MIAs represent weak privacy threats under these conditions.
Membership inference attacks (MIAs) aim to determine whether a data sample was included in a machine learning (ML) model's training set and have become the de facto standard for measuring privacy leakages in ML. We propose an evaluation framework that defines the conditions under which MIAs constitute a genuine privacy threat, and review representative MIAs against it. We find that, under the realistic conditions defined in our framework, MIAs represent weak privacy threats. Thus, relying on them as a privacy metric in ML can lead to an overestimation of risk and to unnecessary sacrifices in model utility as a consequence of employing too strong defenses.