The Cost of Privacy in Asynchronous Differentially-Private Machine Learning
This addresses privacy-preserving collaborative ML for distributed data owners (e.g., in finance/medicine), though it is incremental as it builds on existing differential privacy and asynchronous methods.
The paper tackles the problem of training machine learning models on distributed private datasets with asynchronous communication, developing differentially-private asynchronous algorithms that scale efficiently to many data owners. It proves that the cost of privacy (performance gap from non-private training) has an upper bound inversely proportional to dataset size squared and privacy budget squared, with experiments showing collaboration among over 10 data owners with ≥10,000 records and privacy budget ≥1 yields superior models compared to isolated training.
We consider training machine learning models using Training data located on multiple private and geographically-scattered servers with different privacy settings. Due to the distributed nature of the data, communicating with all collaborating private data owners simultaneously may prove challenging or altogether impossible. In this paper, we develop differentially-private asynchronous algorithms for collaboratively training machine-learning models on multiple private datasets. The asynchronous nature of the algorithms implies that a central learner interacts with the private data owners one-on-one whenever they are available for communication without needing to aggregate query responses to construct gradients of the entire fitness function. Therefore, the algorithm efficiently scales to many data owners. We define the cost of privacy as the difference between the fitness of a privacy-preserving machine-learning model and the fitness of trained machine-learning model in the absence of privacy concerns. We prove that we can forecast the performance of the proposed privacy-preserving asynchronous algorithms. We demonstrate that the cost of privacy has an upper bound that is inversely proportional to the combined size of the training datasets squared and the sum of the privacy budgets squared. We validate the theoretical results with experiments on financial and medical datasets. The experiments illustrate that collaboration among more than 10 data owners with at least 10,000 records with privacy budgets greater than or equal to 1 results in a superior machine-learning model in comparison to a model trained in isolation on only one of the datasets, illustrating the value of collaboration and the cost of the privacy. The number of the collaborating datasets can be lowered if the privacy budget is higher.