Model Evaluation in Medical Datasets Over Time
This addresses the issue of time-agnostic evaluation in medical AI, which is incremental as it provides a new framework for a known bottleneck in model deployment.
The paper tackles the problem of evaluating machine learning models in healthcare where data evolves over time, by introducing the EMDOT framework to assess performance changes across five medical datasets and different training strategies.
Machine learning models deployed in healthcare systems face data drawn from continually evolving environments. However, researchers proposing such models typically evaluate them in a time-agnostic manner, with train and test splits sampling patients throughout the entire study period. We introduce the Evaluation on Medical Datasets Over Time (EMDOT) framework and Python package, which evaluates the performance of a model class over time. Across five medical datasets and a variety of models, we compare two training strategies: (1) using all historical data, and (2) using a window of the most recent data. We note changes in performance over time, and identify possible explanations for these shocks.