A Survey for Federated Learning Evaluations: Goals and Measures
This survey addresses the interdisciplinary evaluation problem for researchers and practitioners in federated learning, but it is incremental as it synthesizes existing work and provides a tool.
The paper tackles the challenge of evaluating federated learning (FL) systems by reviewing evaluation goals and metrics, and introduces FedEval, an open-source platform for standardized assessment of utility, efficiency, and security.
Evaluation is a systematic approach to assessing how well a system achieves its intended purpose. Federated learning (FL) is a novel paradigm for privacy-preserving machine learning that allows multiple parties to collaboratively train models without sharing sensitive data. However, evaluating FL is challenging due to its interdisciplinary nature and diverse goals, such as utility, efficiency, and security. In this survey, we first review the major evaluation goals adopted in the existing studies and then explore the evaluation metrics used for each goal. We also introduce FedEval, an open-source platform that provides a standardized and comprehensive evaluation framework for FL algorithms in terms of their utility, efficiency, and security. Finally, we discuss several challenges and future research directions for FL evaluation.