Distributed Online Learning with Multiple Kernels
This work addresses efficient distributed learning for edge devices in streaming scenarios, though it is incremental as it builds on existing decentralized and kernel methods.
The authors tackled the problem of learning a nonlinear function in a fully decentralized online setting across a network of learners with streaming data, proposing DOMKL, which achieves optimal sublinear regret and matches centralized state-of-the-art performance while keeping data local.
We consider the problem of learning a nonlinear function over a network of learners in a fully decentralized fashion. Online learning is additionally assumed, where every learner receives continuous streaming data locally. This learning model is called a fully distributed online learning (or a fully decentralized online federated learning). For this model, we propose a novel learning framework with multiple kernels, which is named DOMKL. The proposed DOMKL is devised by harnessing the principles of an online alternating direction method of multipliers and a distributed Hedge algorithm. We theoretically prove that DOMKL over T time slots can achieve an optimal sublinear regret, implying that every learner in the network can learn a common function which has a diminishing gap from the best function in hindsight. Our analysis also reveals that DOMKL yields the same asymptotic performance of the state-of-the-art centralized approach while keeping local data at edge learners. Via numerical tests with real datasets, we demonstrate the effectiveness of the proposed DOMKL on various online regression and time-series prediction tasks.