Replicability and Reproducibility of a Schema Evolution Study in Embedded Databases
This work addresses the lack of replication studies in database research, highlighting the effort needed for reproducibility, but it is incremental as it focuses on a specific case study.
The paper tackled the problem of replicating and reproducing a schema evolution study in embedded databases, finding exact results for one out of four applications and close results in two others.
Ascertaining the feasibility of independent falsification or repetition of published results is vital to the scientific process, and replication or reproduction experiments are routinely performed in many disciplines. Unfortunately, such studies are only scarcely available in database research, with few papers dedicated to re-evaluating published results. In this paper, we conduct a case study on replicating and reproducing a study on schema evolution in embedded databases. We obtain exact results for one out of four database applications studied, and come close in two further cases. By reporting results, efforts, and obstacles encountered, we hope to increase appreciation for the substantial efforts required to ensure reproducibility. By discussing minutiae details required for reproducible work, we argue that such important, but often ignored components of scientific work should receive more credit in the evaluation of future research.