Kai Petersen

Ramtin Jabbari, Nauman bin Ali, Kai Petersen

Software process improvement (SPI) is a means to an end, not an end in itself (e.g., a goal is to achieve shorter time to market and not just compliance to a process standard). Therefore, SPI initiatives ought to be streamlined to meet the desired values for an organization. Through a literature review, seven secondary studies aggregating maturity models and assessment frameworks were identified. Furthermore, we identified six proposals for building a new maturity model. We analyzed the existing maturity models for (a) their purpose, structure, guidelines, and (b) the degree to which they explicitly consider values and benefits. Based on this analysis and utilizing the guidelines from the proposals to build maturity models, we have introduced an approach for developing a value-driven approach for SPI. The proposal leveraged the benefits-dependency networks. We argue that our approach enables the following key benefits: (a) as a value-driven approach, it streamlines value-delivery and helps to avoid unnecessary process interventions, (b) as a knowledge-repository, it helps to codify lessons learned i.e. whether adopted practices lead to value realization, and (c) as an internal process maturity assessment tool, it tracks the progress of process realization, which is necessary to monitor progress towards the intended values.

Jefferson Seide Molléri, Kai Petersen, Emilia Mendes

Context: Over the past decade Software Engineering research has seen a steady increase in survey-based studies, and there are several guidelines providing support for those willing to carry out surveys. The need for auditing survey research has been raised in the literature. Checklists have been used to assess different types of empirical studies, such as experiments and case studies. Objective: This paper proposes a checklist to support the design and assessment of survey-based research in software engineering grounded in existing guidelines for survey research. We further evaluated the checklist in the research practice context. Method: To construct the checklist, we systematically aggregated knowledge from 14 methodological papers supporting survey-based research in software engineering. We identified the key stages of the survey process and its recommended practices through thematic analysis and vote counting. To improve our initially designed checklist we evaluated it using a mixed evaluation approach involving experienced researchers. Results: The evaluation provided insights regarding limitations of the checklist in relation to its understanding and objectivity. In particular, 19 of the 38 checklist items were improved according to the feedback received from its evaluation. Finally, a discussion on how to use the checklist and what its implications are for research practice is also provided. Conclusion: The proposed checklist is an instrument suitable for auditing survey reports as well as a support tool to guide ongoing research with regard to the survey design process.

Ahmad Nauman Ghazi, Kai Petersen, Sri Sai Vijay Raj Reddy et al.

Background: The need for empirical investigations in software engineering is growing. Many researchers nowadays, conduct and validate their solutions using empirical research. Survey is one empirical method which enables researchers to collect data from a large population. Main aim of the survey is to generalize the findings. Aims: In this study we aim to identify the problems researchers face during survey design, and mitigation strategies. Method: A literature review as well as semi-structured interviews with nine software engineering researchers were conducted to elicit their views on problems and mitigation strategies. The researchers are all focused on empirical software engineering. Results: We identified 24 problems and 65 strategies, structured according to the survey research process. The most commonly discussed problem was sampling, in particular the ability to obtain a sufficiently large sample. To improve survey instrument design, evaluation and execution recommendations for question formulation and survey pre-testing were given. The importance of involving multiple researchers in the analysis of survey results was stressed. Conclusions: The elicited problems and strategies may serve researchers during the design of their studies. However, it was observed that some strategies were conflicting. This shows that it is important to conduct a trade-off analysis between strategies.

Ahmad Nauman Ghazi, Kai Petersen, Claes Wohlin et al.

Exploratory testing is neither black nor white, but rather a continuum of exploration exists. In this research we propose an approach for decision support helping practitioners to distribute time between different degrees of exploratory testing on that continuum. To make the continuum manageable, five levels have been defined: freestyle testing, high, medium and low degrees of exploration, and scripted testing. The decision support approach is based on the repertory grid technique. The approach has been used in one company. The method for data collection was focus groups. The results showed that the proposed approach aids practitioners in the reflection of what exploratory testing levels to use, and aligns their understanding for priorities of decision criteria and the performance of exploratory testing levels in their contexts. The findings also showed that the participating company, which is currently conducting mostly scripted testing, should spend more time on testing using higher degrees of exploration in comparison to scripted testing.

Ahmad Nauman Ghazi, Ratna Pranathi Garigapati, Kai Petersen

During exploratory testing sessions the tester simultaneously learns, designs and executes tests. The activity is iterative and utilizes the skills of the tester and provides flexibility and creativity.Test charters are used as a vehicle to support the testers during the testing. The aim of this study is to support practitioners in the design of test charters through checklists. We aimed to identify factors allowing practitioners to critically reflect on their designs and contents of test charters to support practitioners in making informed decisions of what to include in test charters. The factors and contents have been elicited through interviews. Overall, 30 factors and 35 content elements have been elicited.

Ahmad Nauman Ghazi, Kai Petersen, Elizabeth Bjarnason et al.

Exploratory testing (ET) is a powerful and efficient way of testing software by integrating design, execution, and analysis of tests during a testing session. ET is often contrasted with scripted testing, and seen as a choice between black and white. We pose that there are different levels of exploratory testing from fully exploratory to fully scripted and propose a scale for the degree of exploration for ET. The degree is defined through levels of ET, which correspond to the way test charters are formulated. We have evaluated the classification through focus groups at four companies and identified factors that influence the level of exploratory testing. The results show that the proposed ET levels have distinguishing characteristics and that the levels can be used as a guide to structure test charters. Our study also indicates that applying a combination of ET levels can be beneficial in achieving effective testing.

Bogdan Marculescu, Simon Poulding, Robert Feldt et al.

Context: Search-based software testing promises to provide users with the ability to generate high-quality test cases, and hence increase product quality, with a minimal increase in the time and effort required. One result that emerged out of a previous study to investigate the application of search-based software testing (SBST) in an industrial setting was the development of the Interactive Search-Based Software Testing (ISBST) system. ISBST allows users to interact with the underlying SBST system, guiding the search and assessing the results. An industrial evaluation indicated that the ISBST system could find test cases that are not created by testers employing manual techniques. The validity of the evaluation was threatened, however, by the low number of participants. Objective: This paper presents a follow-up study, to provide a more rigorous evaluation of the ISBST system. Method: To assess the ISBST system a two-way crossover controlled experiment was conducted with 58 students taking a Verification and Validation course. The NASA Task Load Index (NASA-TLX) is used to assess the workload experienced by the participants in the experiment. Results: The experimental results validated the hypothesis that the ISBST system generates test cases that are not found by the same participants employing manual testing techniques. A follow-up laboratory experiment also investigates the importance of interaction in obtaining the results. In addition to this main result, the subjective workload was assessed for each participant by means of the NASA-TLX tool. The evaluation showed that, while the ISBST system required more effort from the participants, they achieved the same performance. Conclusions: The paper provides evidence that the ISBST system develops test cases that are not found by manual techniques, and that interaction plays an important role in achieving that result.