Mosaic Learning: A Framework for Decentralized Learning with Model Fragmentation
This addresses the challenge of collaborative machine learning in settings where data cannot be centrally hosted, offering a novel approach to improve performance without sacrificing utility or efficiency.
The paper tackles the problem of decentralized learning by introducing Mosaic Learning, a framework that decomposes models into fragments for dissemination across networks, resulting in up to 12 percentage points higher node-level test accuracy compared to a state-of-the-art baseline.
Decentralized learning (DL) enables collaborative machine learning (ML) without a central server, making it suitable for settings where training data cannot be centrally hosted. We introduce Mosaic Learning, a DL framework that decomposes models into fragments and disseminates them independently across the network. Fragmentation reduces redundant communication across correlated parameters and enables more diverse information propagation without increasing communication cost. We theoretically show that Mosaic Learning (i) shows state-of-the-art worst-case convergence rate, and (ii) leverages parameter correlation in an ML model, improving contraction by reducing the highest eigenvalue of a simplified system. We empirically evaluate Mosaic Learning on four learning tasks and observe up to 12 percentage points higher node-level test accuracy compared to epidemic learning (EL), a state-of-the-art baseline. In summary, Mosaic Learning improves DL performance without sacrificing its utility or efficiency, and positions itself as a new DL standard.