The Quest for Reliable Metrics of Responsible AI
This work tackles the problem of unreliable evaluation metrics for responsible AI, which is crucial for developers and researchers aiming to ensure ethical AI practices, though it is incremental as it builds on existing studies.
The paper addresses the lack of robustness and reliability in metrics used to evaluate responsible AI, proposing guidelines based on prior work on fairness metrics for recommender systems to improve metric development across AI applications, including AI in Science.
The development of Artificial Intelligence (AI), including AI in Science (AIS), should be done following the principles of responsible AI. Progress in responsible AI is often quantified through evaluation metrics, yet there has been less work on assessing the robustness and reliability of the metrics themselves. We reflect on prior work that examines the robustness of fairness metrics for recommender systems as a type of AI application and summarise their key takeaways into a set of non-exhaustive guidelines for developing reliable metrics of responsible AI. Our guidelines apply to a broad spectrum of AI applications, including AIS.