Distributed Optimization for Client-Server Architecture with Negative Gradient Weights
This addresses privacy-preserving distributed optimization for machine learning applications, but it appears incremental as it builds on existing multi-parameter-server architectures.
The paper tackles the problem of distributed optimization in a client-server architecture with privacy constraints by proposing a synchronous algorithm for a multi-parameter-server setup, proving it can optimize the overall objective function for arbitrary time-varying topologies with C clients and S servers.
Availability of both massive datasets and computing resources have made machine learning and predictive analytics extremely pervasive. In this work we present a synchronous algorithm and architecture for distributed optimization motivated by privacy requirements posed by applications in machine learning. We present an algorithm for the recently proposed multi-parameter-server architecture. We consider a group of parameter servers that learn a model based on randomized gradients received from clients. Clients are computational entities with private datasets (inducing a private objective function), that evaluate and upload randomized gradients to the parameter servers. The parameter servers perform model updates based on received gradients and share the model parameters with other servers. We prove that the proposed algorithm can optimize the overall objective function for a very general architecture involving $C$ clients connected to $S$ parameter servers in an arbitrary time varying topology and the parameter servers forming a connected network.