Distributed Primal-Dual Optimization for Online Multi-Task Learning
This work addresses communication and efficiency bottlenecks for decentralized online multi-task learning, offering an incremental improvement for scenarios with bandwidth or energy constraints.
The paper tackles the problems of heavy communication and high runtime complexity in online multi-task learning by proposing a distributed primal-dual optimization algorithm that allows geographically separated tasks to synchronize data efficiently. Empirical results show it is highly effective on real-world datasets, with theoretical guarantees of optimal regret.
Conventional online multi-task learning algorithms suffer from two critical limitations: 1) Heavy communication caused by delivering high velocity of sequential data to a central machine; 2) Expensive runtime complexity for building task relatedness. To address these issues, in this paper we consider a setting where multiple tasks are geographically located in different places, where one task can synchronize data with others to leverage knowledge of related tasks. Specifically, we propose an adaptive primal-dual algorithm, which not only captures task-specific noise in adversarial learning but also carries out a projection-free update with runtime efficiency. Moreover, our model is well-suited to decentralized periodic-connected tasks as it allows the energy-starved or bandwidth-constraint tasks to postpone the update. Theoretical results demonstrate the convergence guarantee of our distributed algorithm with an optimal regret. Empirical results confirm that the proposed model is highly effective on various real-world datasets.