Exact Subspace Diffusion for Decentralized Multitask Learning
This work addresses the challenge of enabling effective collaboration among agents with diverse tasks in decentralized settings, representing an incremental advancement in multitask learning methods.
The paper tackles the problem of performance degradation in distributed learning under heterogeneous objectives by developing an exact diffusion algorithm for decentralized multitask learning, achieving improved performance over approximate projection-based alternatives with verified numerical accuracy.
Classical paradigms for distributed learning, such as federated or decentralized gradient descent, employ consensus mechanisms to enforce homogeneity among agents. While these strategies have proven effective in i.i.d. scenarios, they can result in significant performance degradation when agents follow heterogeneous objectives or data. Distributed strategies for multitask learning, on the other hand, induce relationships between agents in a more nuanced manner, and encourage collaboration without enforcing consensus. We develop a generalization of the exact diffusion algorithm for subspace constrained multitask learning over networks, and derive an accurate expression for its mean-squared deviation when utilizing noisy gradient approximations. We verify numerically the accuracy of the predicted performance expressions, as well as the improved performance of the proposed approach over alternatives based on approximate projections.