Finite-Time Consensus Learning for Decentralized Optimization with Nonlinear Gossiping
This work addresses the synchronization problem in decentralized optimization, offering a novel method that enhances performance for applications like deep neural networks, though it is incremental in nature.
The paper tackles the performance gap in decentralized learning by attributing it to poor synchronization among workers, and introduces a nonlinear gossiping framework that achieves finite-time consensus, leading to improved convergence rates validated through extensive benchmarking.
Distributed learning has become an integral tool for scaling up machine learning and addressing the growing need for data privacy. Although more robust to the network topology, decentralized learning schemes have not gained the same level of popularity as their centralized counterparts for being less competitive performance-wise. In this work, we attribute this issue to the lack of synchronization among decentralized learning workers, showing both empirically and theoretically that the convergence rate is tied to the synchronization level among the workers. Such motivated, we present a novel decentralized learning framework based on nonlinear gossiping (NGO), that enjoys an appealing finite-time consensus property to achieve better synchronization. We provide a careful analysis of its convergence and discuss its merits for modern distributed optimization applications, such as deep neural networks. Our analysis on how communication delay and randomized chats affect learning further enables the derivation of practical variants that accommodate asynchronous and randomized communications. To validate the effectiveness of our proposal, we benchmark NGO against competing solutions through an extensive set of tests, with encouraging results reported.