Communication Efficient Distributed Optimization using an Approximate Newton-type Method
This addresses communication bottlenecks in distributed machine learning, offering a solution for large-scale stochastic optimization and learning problems.
The paper tackles the problem of communication efficiency in distributed optimization by introducing a novel Newton-type method, achieving a linear convergence rate for quadratic objectives that improves with data size and requires essentially constant iterations under reasonable assumptions.
We present a novel Newton-type method for distributed optimization, which is particularly well suited for stochastic optimization and learning problems. For quadratic objectives, the method enjoys a linear rate of convergence which provably \emph{improves} with the data size, requiring an essentially constant number of iterations under reasonable assumptions. We provide theoretical and empirical evidence of the advantages of our method compared to other approaches, such as one-shot parameter averaging and ADMM.