Omar S. Gómez

Adrian Santos, Omar Gomez, Natalia Juristo

Context: Families of experiments (i.e., groups of experiments with the same goal) are on the rise in Software Engineering (SE). Selecting unsuitable aggregation techniques to analyze families may undermine their potential to provide in-depth insights from experiments' results. Objectives: Identifying the techniques used to aggregate experiments' results within families in SE. Raising awareness of the importance of applying suitable aggregation techniques to reach reliable conclusions within families. Method: We conduct a systematic mapping study (SMS) to identify the aggregation techniques used to analyze families of experiments in SE. We outline the advantages and disadvantages of each aggregation technique according to mature experimental disciplines such as medicine and pharmacology. We provide preliminary recommendations to analyze and report families of experiments in view of families' common limitations with regard to joint data analysis. Results: Several aggregation techniques have been used to analyze SE families of experiments, including Narrative synthesis, Aggregated Data (AD), Individual Participant Data (IPD) mega-trial or stratified, and Aggregation of p-values. The rationale used to select aggregation techniques is rarely discussed within families. Families of experiments are commonly analyzed with unsuitable aggregation techniques according to the literature of mature experimental disciplines. Conclusion: Data analysis' reporting practices should be improved to increase the reliability and transparency of joint results. AD and IPD stratified appear to be suitable to analyze SE families of experiments.

Omar S. Gómez

Agile software development has been widespread adopted. One well-known agile approach is eXtreme Programming (XP) where pair programming (PP) is a relevant practice. Although various aspects of PP have been studied, we have not found, under a traditional model of PP, studies that examine the impact of using an IDE tool support. In an attempt to obtain a better understanding of the impact of using an IDE, we present the results of a controlled experiment that expose the influence on quality, measured as the number of defects injected per hour, and cost, measured as the time necessary to complete programming assignments, of pair and solo programming with and without the use of an IDE. For quality, our findings suggest that the use of an IDE results in significantly higher defect injection rates (for both pairs and solos) when the programming assignment is not very complicated. Nevertheless, defect injection rates seem to decrease when pairs work on more complicated programming assignments irrespective of the tool support used. For cost, the programming assignment significantly affects the time necessary to complete the assignment. Finally, both aspects (quality and cost) are affected in a similar manner when either pair or solo programming is used.

Omar S. Gómez, Raúl A. Aguilar, Juan P. Ucán

The main objective of this work is to examine possible effects of using freshman student subjects in software engineering experiments. Particularly in this work we report the effectiveness measured as percentage of observed and observable defects of two software testing techniques: Black-box and white-box. Regarding observed defects, both techniques show an effectiveness around of 4%. With respect of observable defects by test cases, black-box testing is slightly more effective (21%) than white-box testing (16%), although this difference is not significant. We observe a considerable lack of technical skills of subjects for applying both software testing techniques. Due to observed findings, we suggest to employ students with more technical skills for carrying out software engineering experiments. ----- El objetivo de este trabajo se centra en investigar los efectos que conlleva realizar experimentos en ingeniería de software (IS) empleando como sujetos experimentales a estudiantes de pregrado cursando su primer año de estudios de la carrera en ingeniería de software. De manera particular en este trabajo se investiga la efectividad medida en porcentaje de defectos observados y observables de las técnicas de prueba de software funcional (caja negra) y estructural (caja blanca). Con respecto a los defectos observados por los sujetos, ambas técnicas obtuvieron una efectividad del 4%. Con respecto a los defectos observables por los casos de prueba, la técnica funcional es ligeramente superior (21%) que la técnica estructural (16%), aunque esta diferencia no es significativa. Se observa un nivel de inexperiencia considerable en los sujetos para aplicar las técnicas. Dado los hallazgos encontrados, se sugiere emplear sujetos de pregrado con un nivel mayor de experiencia.

Omar S. Gómez, José L. Batún, Raúl A. Aguilar

This paper presents an experience report about an experiment that evaluates duration and effort of pair and solo programming. The experiment was performed as part of a course on Design of Experiments (DOE) in Software Engineering (SE) at Autonomous University of Yucatan (UADY). A total of 21 junior student subjects enrolled in the bachelor's degree program in SE participated in the experiment. During the experiment, subjects (7 pairs and 7 solos) wrote two small programs in two sessions. Results show a significant difference (at alpha=0.1) in favor of pair programming regarding duration (28% decrease), and a significant difference (at alpha=0.1) in favor of solo programming with respect to effort (30% decrease). With only a difference of 1%, our results regarding duration and effort are practically the same as those reported by Nosek in 1998.