Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering
This addresses the challenge of enabling unsupervised learning in federated settings for applications where labeled data is scarce, though it is an incremental improvement over prior centralized self-supervised extensions.
The paper tackles the problem of unsupervised federated learning, which lacks robust methods for handling statistical and system heterogeneity, by proposing Orchestra, a technique that uses globally consistent clustering to partition client data. It shows that Orchestra outperforms alternatives in various conditions, such as with 10-100 clients and up to 50% participation ratios, achieving higher accuracy in linear probe evaluations.
Federated learning is generally used in tasks where labels are readily available (e.g., next word prediction). Relaxing this constraint requires design of unsupervised learning techniques that can support desirable properties for federated training: robustness to statistical/systems heterogeneity, scalability with number of participants, and communication efficiency. Prior work on this topic has focused on directly extending centralized self-supervised learning techniques, which are not designed to have the properties listed above. To address this situation, we propose Orchestra, a novel unsupervised federated learning technique that exploits the federation's hierarchy to orchestrate a distributed clustering task and enforce a globally consistent partitioning of clients' data into discriminable clusters. We show the algorithmic pipeline in Orchestra guarantees good generalization performance under a linear probe, allowing it to outperform alternative techniques in a broad range of conditions, including variation in heterogeneity, number of clients, participation ratio, and local epochs.