Subgroup Validity in Machine Learning for Echocardiogram Data
This addresses fairness and reliability concerns in deploying cardiac ultrasound models for diverse patient populations, but it is incremental as it highlights existing deficiencies without proposing new solutions.
The paper tackled the problem of subgroup validity in machine learning for echocardiogram data by analyzing open datasets and finding insufficient demographic reporting and patient counts, with exploratory analysis showing no evidence for validity across sex, racial, and ethnic subgroups.
Echocardiogram datasets enable training deep learning models to automate interpretation of cardiac ultrasound, thereby expanding access to accurate readings of diagnostically-useful images. However, the gender, sex, race, and ethnicity of the patients in these datasets are underreported and subgroup-specific predictive performance is unevaluated. These reporting deficiencies raise concerns about subgroup validity that must be studied and addressed before model deployment. In this paper, we show that current open echocardiogram datasets are unable to assuage subgroup validity concerns. We improve sociodemographic reporting for two datasets: TMED-2 and MIMIC-IV-ECHO. Analysis of six open datasets reveals no consideration of gender-diverse patients and insufficient patient counts for many racial and ethnic groups. We further perform an exploratory subgroup analysis of two published aortic stenosis detection models on TMED-2. We find insufficient evidence for subgroup validity for sex, racial, and ethnic subgroups. Our findings highlight that more data for underrepresented subgroups, improved demographic reporting, and subgroup-focused analyses are needed to prove subgroup validity in future work.