Exploring Scaling Laws for EHR Foundation Models
This work addresses the need for resource-efficient training strategies for EHR foundation models, which could transform clinical prediction tasks and personalized healthcare, though it is incremental as it extends known scaling principles to a new domain.
The study tackled the problem of applying scaling laws, which predict performance gains in large language models, to electronic health records (EHRs) by training transformer models on MIMIC-IV data, and found consistent scaling patterns like parabolic IsoFLOPs curves and power-law relationships that demonstrate analogous behavior to LLMs.
The emergence of scaling laws has profoundly shaped the development of large language models (LLMs), enabling predictable performance gains through systematic increases in model size, dataset volume, and compute. Yet, these principles remain largely unexplored in the context of electronic health records (EHRs) -- a rich, sequential, and globally abundant data source that differs structurally from natural language. In this work, we present the first empirical investigation of scaling laws for EHR foundation models. By training transformer architectures on patient timeline data from the MIMIC-IV database across varying model sizes and compute budgets, we identify consistent scaling patterns, including parabolic IsoFLOPs curves and power-law relationships between compute, model parameters, data size, and clinical utility. These findings demonstrate that EHR models exhibit scaling behavior analogous to LLMs, offering predictive insights into resource-efficient training strategies. Our results lay the groundwork for developing powerful EHR foundation models capable of transforming clinical prediction tasks and advancing personalized healthcare.