Federated Unsupervised Semantic Segmentation
It addresses the challenge of decentralized, label-free semantic segmentation for applications like medical imaging or autonomous driving, representing an incremental advancement by adapting existing USS methods to federated settings.
This paper tackles the problem of performing unsupervised semantic image segmentation in a federated learning setting, where data is distributed across clients with heterogeneous distributions and no labels, by proposing the FUSS framework that introduces novel federation strategies for feature and prototype alignment, resulting in consistent outperformance over local-only training and classical FL extensions on benchmark and real-world datasets.
This work explores the application of Federated Learning (FL) in Unsupervised Semantic image Segmentation (USS). Recent USS methods extract pixel-level features using frozen visual foundation models and refine them through self-supervised objectives that encourage semantic grouping. These features are then grouped to semantic clusters to produce segmentation masks. Extending these ideas to federated settings requires feature representation and cluster centroid alignment across distributed clients -- an inherently difficult task under heterogeneous data distributions in the absence of supervision. To address this, we propose FUSS Federated Unsupervised image Semantic Segmentation) which is, to our knowledge, the first framework to enable fully decentralized, label-free semantic segmentation training. FUSS introduces novel federation strategies that promote global consistency in feature and prototype space, jointly optimizing local segmentation heads and shared semantic centroids. Experiments on both benchmark and real-world datasets, including binary and multi-class segmentation tasks, show that FUSS consistently outperforms local-only client trainings as well as extensions of classical FL algorithms under varying client data distributions. To support reproducibility, full code will be released upon manuscript acceptance.