One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Models
For researchers developing EHR foundation models, this work addresses the critical bottleneck of capturing irregular temporal structure and continuous measurements, yielding more generalizable representations.
The paper proposes ORA, a marked time-to-event pretraining objective for EHR foundation models that jointly models event timing and associated measurements, consistently outperforming next-token prediction and other losses across multiple datasets and tasks, including regression and time-to-event prediction.
Clinical events captured in Electronic Health Records (EHR) are irregularly sampled and may consist of a mixture of discrete events and numerical measurements, such as laboratory values or treatment dosages. The sequential nature of EHR, analogous to natural language, has motivated the use of next-token prediction to train prior EHR Foundation Models (FMs) over events. However, this training fails to capture the full structure of EHR. When a given event occurs must be captured, but the event value (abnormal lab) also modulates the likelihood of other clinical events. Most existing EHR FMs do not jointly model this likelihood and are unable to capture the full observation process, impacting downstream capabilities. We propose ORA, a marked time-to-event pretraining objective that jointly models event timing and associated measurements. Across multiple datasets, downstream tasks, and model backbones, this objective consistently yields more generalizable representations than next-token prediction and pretraining losses that ignore continuous measurements. Importantly, the proposed objective yields improvements beyond traditional classification evaluation, including better regression and time-to-event prediction. Beyond introducing a new family of FMs, our ablations suggest a broader takeaway: pretraining objectives that account for EHR structure are critical for expanding downstream capabilities and generalizability.