Decoupled Contrastive Learning for Federated Learning
This addresses data heterogeneity in federated learning for distributed machine learning applications, offering an incremental improvement over existing contrastive learning methods.
The paper tackles performance degradation in federated learning due to data heterogeneity by introducing Decoupled Contrastive Learning for Federated Learning (DCFL), which decouples contrastive loss into alignment and uniformity components to avoid asymptotic assumptions, achieving stronger alignment and uniformity and outperforming state-of-the-art methods on benchmarks like CIFAR-10, CIFAR-100, and Tiny-ImageNet.
Federated learning is a distributed machine learning paradigm that allows multiple participants to train a shared model by exchanging model updates instead of their raw data. However, its performance is degraded compared to centralized approaches due to data heterogeneity across clients. While contrastive learning has emerged as a promising approach to mitigate this, our theoretical analysis reveals a fundamental conflict: its asymptotic assumptions of an infinite number of negative samples are violated in finite-sample regime of federated learning. To address this issue, we introduce Decoupled Contrastive Learning for Federated Learning (DCFL), a novel framework that decouples the existing contrastive loss into two objectives. Decoupling the loss into its alignment and uniformity components enables the independent calibration of the attraction and repulsion forces without relying on the asymptotic assumptions. This strategy provides a contrastive learning method suitable for federated learning environments where each client has a small amount of data. Our experimental results show that DCFL achieves stronger alignment between positive samples and greater uniformity between negative samples compared to existing contrastive learning methods. Furthermore, experimental results on standard benchmarks, including CIFAR-10, CIFAR-100, and Tiny-ImageNet, demonstrate that DCFL consistently outperforms state-of-the-art federated learning methods.