Evaluating subgroup disparity using epistemic uncertainty in mammography
This work addresses the need for accountability and generalizability in clinical ML deployments by providing a method to evaluate disparities, though it is incremental as it applies existing uncertainty concepts to a specific healthcare domain.
The paper tackled the problem of detecting subgroup disparities in machine learning models for healthcare by using epistemic uncertainty to evaluate disparities in race and scanner subgroups for breast density assessment on 108,190 mammograms. The result showed that the choice of uncertainty quantification metric significantly affects subgroup-level performance, even when aggregate performance is comparable.
As machine learning (ML) continue to be integrated into healthcare systems that affect clinical decision making, new strategies will need to be incorporated in order to effectively detect and evaluate subgroup disparities to ensure accountability and generalizability in clinical workflows. In this paper, we explore how epistemic uncertainty can be used to evaluate disparity in patient demographics (race) and data acquisition (scanner) subgroups for breast density assessment on a dataset of 108,190 mammograms collected from 33 clinical sites. Our results show that even if aggregate performance is comparable, the choice of uncertainty quantification metric can significantly the subgroup level. We hope this analysis can promote further work on how uncertainty can be leveraged to increase transparency of machine learning applications for clinical deployment.