Gradient Disaggregation: Breaking Privacy in Federated Learning by Reconstructing the User Participant Matrix
This exposes a critical privacy vulnerability in federated learning systems, allowing servers to break secure aggregation and compromise individual user data, which is a major concern for applications relying on data anonymity.
The paper tackles the insecurity of aggregated model updates in federated learning by showing that an untrusted central server can disaggregate user updates from sums, enabling recovery of private training data via gradient inference attacks. The attack successfully handles up to thousands of participants and significantly improves inference capabilities, violating user anonymity and privacy.
We show that aggregated model updates in federated learning may be insecure. An untrusted central server may disaggregate user updates from sums of updates across participants given repeated observations, enabling the server to recover privileged information about individual users' private training data via traditional gradient inference attacks. Our method revolves around reconstructing participant information (e.g: which rounds of training users participated in) from aggregated model updates by leveraging summary information from device analytics commonly used to monitor, debug, and manage federated learning systems. Our attack is parallelizable and we successfully disaggregate user updates on settings with up to thousands of participants. We quantitatively and qualitatively demonstrate significant improvements in the capability of various inference attacks on the disaggregated updates. Our attack enables the attribution of learned properties to individual users, violating anonymity, and shows that a determined central server may undermine the secure aggregation protocol to break individual users' data privacy in federated learning.