Stochastic Distributed Optimization for Machine Learning from Decentralized Features
This addresses data privacy and locality challenges in distributed machine learning for applications where features are inherently decentralized, though it is incremental as it builds on existing SGD and distributed optimization methods.
The paper tackles the problem of training machine learning models when features for the same samples are decentralized across different parties, proposing an asynchronous SGD algorithm that preserves data confidentiality without sharing features or local parameters. Experimental results on a real-world dataset with 5 million samples and 8700 features demonstrate the system's effectiveness and efficiency.
Distributed machine learning has been widely studied in the literature to scale up machine learning model training in the presence of an ever-increasing amount of data. We study distributed machine learning from another perspective, where the information about the training same samples are inherently decentralized and located on different parities. We propose an asynchronous stochastic gradient descent (SGD) algorithm for such a feature distributed machine learning (FDML) problem, to jointly learn from decentralized features, with theoretical convergence guarantees under bounded asynchrony. Our algorithm does not require sharing the original feature data or even local model parameters between parties, thus preserving a high level of data confidentiality. We implement our algorithm for FDML in a parameter server architecture. We compare our system with fully centralized training (which violates data locality requirements) and training only based on local features, through extensive experiments performed on a large amount of data from a real-world application, involving 5 million samples and $8700$ features in total. Experimental results have demonstrated the effectiveness and efficiency of the proposed FDML system.