Federated learning, ethics, and the double black box problem in medical AI
This addresses ethical concerns for medical AI practitioners and patients, but is incremental as it builds on existing critiques of opacity in AI.
The paper tackles the underexamined ethical risks of federated learning in medical AI, arguing that it creates a 'federation opacity' leading to a double black box problem, and concludes by highlighting challenges for ethical feasibility.
Federated learning (FL) is a machine learning approach that allows multiple devices or institutions to collaboratively train a model without sharing their local data with a third-party. FL is considered a promising way to address patient privacy concerns in medical artificial intelligence. The ethical risks of medical FL systems themselves, however, have thus far been underexamined. This paper aims to address this gap. We argue that medical FL presents a new variety of opacity -- federation opacity -- that, in turn, generates a distinctive double black box problem in healthcare AI. We highlight several instances in which the anticipated benefits of medical FL may be exaggerated, and conclude by highlighting key challenges that must be overcome to make FL ethically feasible in medicine.