Measuring the Stability of EHR- and EKG-based Predictive Models
This addresses the issue of unreliable clinical predictions for healthcare providers, but it is incremental as it focuses on testing and comparing existing data sources rather than introducing a new method.
The paper tackled the problem of poor generalization in predictive models built from electronic health records (EHRs) due to varying health-seeking behaviors across patient populations, proposing two tests to measure model stability and showing that EKG-based models are more stable than EHR-based models across different populations.
Databases of electronic health records (EHRs) are increasingly used to inform clinical decisions. Machine learning methods can find patterns in EHRs that are predictive of future adverse outcomes. However, statistical models may be built upon patterns of health-seeking behavior that vary across patient subpopulations, leading to poor predictive performance when training on one patient population and predicting on another. This note proposes two tests to better measure and understand model generalization. We use these tests to compare models derived from two data sources: (i) historical medical records, and (ii) electrocardiogram (EKG) waveforms. In a predictive task, we show that EKG-based models can be more stable than EHR-based models across different patient populations.