Improving Software Engineering Research through Experimentation Workbenches
This addresses the difficulty of replicating experiments in software engineering research, offering a potential solution for researchers, though it appears incremental as it builds on existing ideas.
The paper tackles the problem of insufficient support for experimentation in software engineering research by proposing the concept of experimentation workbenches, which aim to facilitate experiments and improve reproducibility and repeatability, as illustrated through a scenario and case study.
Experimentation with software prototypes plays a fundamental role in software engineering research. In contrast to many other scientific disciplines, however, explicit support for this key activity in software engineering is relatively small. While some approaches to improve this situation have been proposed by the software engineering community, experiments are still very difficult and sometimes impossible to replicate. In this paper, we propose the concept of an experimentation workbench as a means of explicit support for experimentation in software engineering research. In particular, we discuss core requirements that an experimentation workbench should satisfy in order to qualify as such and to offer a real benefit for researchers. Beyond their core benefits for experimentation, we stipulate that experimentation workbenches will also have benefits in regard to reproducibility and repeatability of software engineering research. Further, we illustrate this concept with a scenario and a case study, and describe relevant challenges as well as our experience with experimentation workbenches.