Defining Replicability of Prediction Rules
It addresses the need for systematic assessment of replicability in machine learning, which is an incremental contribution to methodological guidelines.
The paper tackles the problem of defining replicability for prediction rules, proposing a multi-agent framework to interpret 'consistent results' across studies and recovering existing practical approaches as special cases.
In this article I propose an approach for defining replicability for prediction rules. Motivated by a recent NAS report, I start from the perspective that replicability is obtaining consistent results across studies suitable to address the same prediction question, each of which has obtained its own data. I then discuss concept and issues in defining key elements of this statement. I focus specifically on the meaning of "consistent results" in typical utilization contexts, and propose a multi-agent framework for defining replicability, in which agents are neither partners nor adversaries. I recover some of the prevalent practical approaches as special cases. I hope to provide guidance for a more systematic assessment of replicability in machine learning.