Considerations and Pitfalls in Controlled Experiments on Code Comprehension
This work is incremental, focusing on methodological improvements for researchers in software engineering and program comprehension.
The paper addresses the challenges in designing controlled experiments for code comprehension by identifying potential problems related to subjects, code, tasks, and performance metrics, aiming to promote sound experimental methodology.
Understanding program code is a complicated endeavor. As such, myriad different factors can influence the outcome. Investigations of program comprehension, and in particular those using controlled experiments, have to take these factors into account. In order to promote the development and use of sound experimental methodology, we discuss potential problems with regard to the experimental subjects, the code they work on, the tasks they are asked to perform, and the metrics for their performance.