Elizabeth Bjarnason

Markus Borg, Iben Lennerstad, Rasmus Ros et al.

Abundant data is the key to successful machine learning. However, supervised learning requires annotated data that are often hard to obtain. In a classification task with limited resources, Active Learning (AL) promises to guide annotators to examples that bring the most value for a classifier. AL can be successfully combined with self-training, i.e., extending a training set with the unlabelled examples for which a classifier is the most certain. We report our experiences on using AL in a systematic manner to train an SVM classifier for Stack Overflow posts discussing performance of software components. We show that the training examples deemed as the most valuable to the classifier are also the most difficult for humans to annotate. Despite carefully evolved annotation criteria, we report low inter-rater agreement, but we also propose mitigation strategies. Finally, based on one annotator's work, we show that self-training can improve the classification accuracy. We conclude the paper by discussing implication for future text miners aspiring to use AL and self-training.

Ahmad Nauman Ghazi, Kai Petersen, Claes Wohlin et al.

Exploratory testing is neither black nor white, but rather a continuum of exploration exists. In this research we propose an approach for decision support helping practitioners to distribute time between different degrees of exploratory testing on that continuum. To make the continuum manageable, five levels have been defined: freestyle testing, high, medium and low degrees of exploration, and scripted testing. The decision support approach is based on the repertory grid technique. The approach has been used in one company. The method for data collection was focus groups. The results showed that the proposed approach aids practitioners in the reflection of what exploratory testing levels to use, and aligns their understanding for priorities of decision criteria and the performance of exploratory testing levels in their contexts. The findings also showed that the participating company, which is currently conducting mostly scripted testing, should spend more time on testing using higher degrees of exploration in comparison to scripted testing.

Ahmad Nauman Ghazi, Kai Petersen, Elizabeth Bjarnason et al.

Exploratory testing (ET) is a powerful and efficient way of testing software by integrating design, execution, and analysis of tests during a testing session. ET is often contrasted with scripted testing, and seen as a choice between black and white. We pose that there are different levels of exploratory testing from fully exploratory to fully scripted and propose a scale for the degree of exploration for ET. The degree is defined through levels of ET, which correspond to the way test charters are formulated. We have evaluated the classification through focus groups at four companies and identified factors that influence the level of exploratory testing. The results show that the proposed ET levels have distinguishing characteristics and that the levels can be used as a guide to structure test charters. Our study also indicates that applying a combination of ET levels can be beneficial in achieving effective testing.

Michael Felderer, Elizabeth Bjarnason, Markus Borg et al.

The main objective of the RET workshop was to explore the interaction of Requirements Engineering (RE) and Testing, i.e. RET, in research and industry, and the challenges that result from this interaction. While much work has been done in the respective fields of requirements engineering and testing, there exists much more than can be done to understand the connection between the processes of RE and of testing.