Distributed Inexact Damped Newton Method: Data Partitioning and Load-Balancing
This work addresses scalability problems in distributed optimization for large-scale machine learning applications, presenting incremental improvements to an existing method.
The paper tackles the scalability issues of the DiSCO algorithm for distributed optimization by proposing modifications to reduce communications, improve load-balancing, and enhance computational efficiency, achieving these improvements as demonstrated in numerical experiments on a 273GB dataset.
In this paper we study inexact dumped Newton method implemented in a distributed environment. We start with an original DiSCO algorithm [Communication-Efficient Distributed Optimization of Self-Concordant Empirical Loss, Yuchen Zhang and Lin Xiao, 2015]. We will show that this algorithm may not scale well and propose an algorithmic modifications which will lead to less communications, better load-balancing and more efficient computation. We perform numerical experiments with an regularized empirical loss minimization instance described by a 273GB dataset.