FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records
This work addresses the problem of evaluating foundation models for healthcare applications, providing guidance for developers, but it is incremental as it focuses on evaluation rather than new model development.
The authors tackled the lack of consensus on clinical utility of foundation models for structured EHR data by proposing a suite of clinically meaningful tasks and evaluating state-of-the-art models on 5 million patients across 14 tasks, measuring accuracy, calibration, and subpopulation performance.
Foundation models hold significant promise in healthcare, given their capacity to extract meaningful representations independent of downstream tasks. This property has enabled state-of-the-art performance across several clinical applications trained on structured electronic health record (EHR) data, even in settings with limited labeled data, a prevalent challenge in healthcare. However, there is little consensus on these models' potential for clinical utility due to the lack of desiderata of comprehensive and meaningful tasks and sufficiently diverse evaluations to characterize the benefit over conventional supervised learning. To address this gap, we propose a suite of clinically meaningful tasks spanning patient outcomes, early prediction of acute and chronic conditions, including desiderata for robust evaluations. We evaluate state-of-the-art foundation models on EHR data consisting of 5 million patients from Columbia University Irving Medical Center (CUMC), a large urban academic medical center in New York City, across 14 clinically relevant tasks. We measure overall accuracy, calibration, and subpopulation performance to surface tradeoffs based on the choice of pre-training, tokenization, and data representation strategies. Our study aims to advance the empirical evaluation of structured EHR foundation models and guide the development of future healthcare foundation models.