Lessons Learned: Reproducibility, Replicability, and When to Stop
This provides a practical tool for early career researchers to incorporate prior work and make informed claims, though it is incremental as it builds on existing reproducibility concepts.
The authors tackled the lack of guidance for reproducing and replicating external studies by developing a two-dimensional framework based on dataset, metrics, and model, which helps researchers assess their work and inform claims, particularly in atmospheric sciences.
While extensive guidance exists for ensuring the reproducibility of one's own study, there is little discussion regarding the reproduction and replication of external studies within one's own research. To initiate this discussion, drawing lessons from our experience reproducing an operational product for predicting tropical cyclogenesis, we present a two-dimensional framework to offer guidance on reproduction and replication. Our framework, representing model fitting on one axis and its use in inference on the other, builds upon three key aspects: the dataset, the metrics, and the model itself. By assessing the trajectories of our studies on this 2D plane, we can better inform the claims made using our research. Additionally, we use this framework to contextualize the utility of benchmark datasets in the atmospheric sciences. Our two-dimensional framework provides a tool for researchers, especially early career researchers, to incorporate prior work in their own research and to inform the claims they can make in this context.