Benchmarking Algorithms for Federated Domain Generalization
This provides a benchmark for researchers working on federated learning with domain generalization, though it is incremental as it builds on existing domain generalization and federated learning work.
The authors tackled the problem of evaluating algorithms for federated domain generalization by creating a benchmark methodology that controls client number and heterogeneity, then applied it to 14 methods. Their results showed significant performance gaps persist, especially with many clients, high heterogeneity, or realistic datasets.
While prior domain generalization (DG) benchmarks consider train-test dataset heterogeneity, we evaluate Federated DG which introduces federated learning (FL) specific challenges. Additionally, we explore domain-based heterogeneity in clients' local datasets - a realistic Federated DG scenario. Prior Federated DG evaluations are limited in terms of the number or heterogeneity of clients and dataset diversity. To address this gap, we propose an Federated DG benchmark methodology that enables control of the number and heterogeneity of clients and provides metrics for dataset difficulty. We then apply our methodology to evaluate 14 Federated DG methods, which include centralized DG methods adapted to the FL context, FL methods that handle client heterogeneity, and methods designed specifically for Federated DG. Our results suggest that despite some progress, there remain significant performance gaps in Federated DG particularly when evaluating with a large number of clients, high client heterogeneity, or more realistic datasets. Please check our extendable benchmark code here: https://github.com/inouye-lab/FedDG_Benchmark.