FeDa4Fair: Client-Level Federated Datasets for Fairness Evaluation
This work provides a framework for researchers to better evaluate fairness in FL, addressing heterogeneous biases across clients, though it is incremental as it focuses on benchmarking rather than new algorithmic solutions.
The paper tackles the problem of evaluating fairness in federated learning (FL) by addressing the limitations of current methods that ignore heterogeneous client biases and only assess fairness at the server level, resulting in the introduction of a benchmarking framework with datasets and tools for robust fairness evaluation.
Federated Learning (FL) enables collaborative model training across multiple clients without sharing clients' private data. However, the diverse and often conflicting biases present across clients pose significant challenges to model fairness. Current fairness-enhancing FL solutions often fall short, as they typically mitigate biases for a single, usually binary, sensitive attribute, while ignoring the heterogeneous fairness needs that exist in real-world settings. Moreover, these solutions often evaluate unfairness reduction only on the server side, hiding persistent unfairness at the individual client level. To support more robust and reproducible fairness research in FL, we introduce a comprehensive benchmarking framework for fairness-aware FL at both the global and client levels. Our contributions are three-fold: (1) We introduce \fairdataset, a library to create tabular datasets tailored to evaluating fair FL methods under heterogeneous client bias; (2) we release four bias-heterogeneous datasets and corresponding benchmarks to compare fairness mitigation methods in a controlled environment; (3) we provide ready-to-use functions for evaluating fairness outcomes for these datasets.