Pursuing a Prospective Perspective
This addresses a methodological gap for researchers and practitioners in machine learning, though it appears incremental as it builds on existing validation concepts.
The paper argues that retrospective testing of predictive models is insufficient for real-world deployment, advocating for prospective validation to enable meaningful comparisons and improve reproducibility.
Retrospective testing of predictive models does not consider the real-world context in which models are deployed. Prospective validation, on the other hand, enables meaningful comparisons between data generation processes by incorporating trained models and considering the subjective decisions that affect reproducibility. Prospective experiments are essential for consistent progress in modeling.