Austen Rainer

Austen Rainer, Catherine Menon

Working with stories and working with computations require very different modes of thought. We call the first mode "story-thinking" and the second "computational-thinking". The aim of this curiosity-driven paper is to explore the nature of these two modes of thinking, and to do so in relation to programming, including software engineering as programming-in-the-large. We suggest that story-thinking and computational-thinking may be understood as two ways of attending to the world, and that each both contributes and neglects the world, though in different ways and for different ends. We formulate two fundamental problems, i.e., the problem of "neglectful representations" and the problem of oppositional ways of thinking. We briefly suggest two ways in which these problems might be tackled and identify candidate hypotheses about the current state of the world, one assertion about a possible future state, and several research questions for future research.

Austen Rainer, Claes Wohlin

Context: Software practitioners are a primary provider of information for field studies in software engineering. Research typically recruits practitioners through some kind of sampling. But sampling may not in itself recruit credible participants. Objectives: To propose and demonstrate a framework for recruiting professional practitioners as credible participants in field studies of software engineering. Method: We review existing guidelines, checklists and other advisory sources on recruiting participants for field studies. We develop a framework, partly based on our prior research and on the research of others. We search for and select three exemplar studies (a case study, an interview study and a survey study) and use those to demonstrate the application of the framework. Results: Whilst existing guidelines etc. recognise the importance of recruiting participants, there is limited guidance on how to recruit the right participants. Our demonstration of the framework with three exemplars shows that at least some members of the research community are aware of the need to carefully recruit participants. Conclusions: The framework provides a new perspective for thinking about the recruitment of credible practitioners for field studies of software engineering. In particular, the framework identifies a number of characteristics not explicitly addressed by existing guidelines.

Ashley Williams, Matthew Shardlow, Austen Rainer

Blogs are a source of grey literature which are widely adopted by software practitioners for disseminating opinion and experience. Analysing such articles can provide useful insights into the state-of-practice for software engineering research. However, there are challenges in identifying higher quality content from the large quantity of articles available. Credibility assessment can help in identifying quality content, though there is a lack of existing corpora. Credibility is typically measured through a series of conceptual criteria, with 'argumentation' and 'evidence' being two important criteria. We create a corpus labelled for argumentation and evidence that can aid the credibility community. The corpus consists of articles from the blog of a single software practitioner and is publicly available. Three annotators label the corpus with a series of conceptual credibility criteria, reaching an agreement of 0.82 (Fleiss' Kappa). We present preliminary analysis of the corpus by using it to investigate the identification of claim sentences (one of our ten labels). We train four systems (Bert, KNN, Decision Tree and SVM) using three feature sets (Bag of Words, Topic Modelling and InferSent), achieving an F1 score of 0.64 using InferSent and a Linear SVM. Our preliminary results are promising, indicating that the corpus can help future studies in detecting the credibility of grey literature. Future research will investigate the degree to which the sentence level annotations can infer the credibility of the overall document.

Austen Rainer

BACKGROUND: Software engineering is a human activity. People naturally make sense of their activities and experience through storytelling. But storytelling does not appear to have been properly studied by software engineering research. AIM: We explore the question: what contribution can storytelling make to human--centric software engineering research? METHOD: We define concepts, identify types of story and their purposes, outcomes and effects, briefly review prior literature, identify several contributions and propose next steps. RESULTS: Storytelling can, amongst other contributions, contribute to data collection, data analyses, ways of knowing, research outputs, interventions in practice, and advocacy, and can integrate with evidence and arguments. Like all methods, storytelling brings risks. These risks can be managed. CONCLUSION: Storytelling provides a potential counter--balance to abstraction, and an approach to retain and honour human meaning in software engineering.

Austen Rainer, Ashley Williams

Background: Blog posts are frequently used by software practitioners to share information about their practice. Blog posts therefore provide a potential source of evidence for software engineering (SE) research. The use of blog posts as evidence for research appears contentious amongst some SE researchers. Objective: To better understand the actual and perceived value of blog posts as evidence for SE research, and to develop guidance for SE researchers on the use of blog posts as evidence. Method: We further analyse responses from a previously conducted attitudinal survey of 44 software engineering researchers. We conduct a heatmap analysis, simple statistical analysis, and a thematic analysis. Results: We find no clear consensus from respondents on researchers' attitudes to the credibility of blog posts, or on a standard set of criteria to evaluate blog-post credibility. We show that some of the responses to the survey exhibit characteristics similar to the content of blog posts, e.g., asserting prior beliefs as claims, with no citations and little supporting rationale. We illustrate our insights with ~60 qualitative examples from the survey ~40% of the total responses. We complement our quantitative and qualitative analyses with preliminary checklists to guide SE researchers. Conclusion: Blog posts are relevant to research because they are written by software practitioners describing their practice and experience. But evaluating the credibility of blog posts, so as to select the higher-quality content, remains an ongoing challenge. The quantitative and qualitative results, with the proposed checklists, are intended to stimulate reflection and action in the research community on the role of blog posts as evidence in software engineering research. Finally, our findings on researchers' attitudes to blog posts also provide more general insights into researchers' values for SE research.

Austen Rainer, Ashley Williams, Vahid Garousi et al.

Background: Retrieving and mining practitioners' self--reports of their professional experience of software practice could provide valuable evidence for research. We are, however, unaware of any existing reviews of research conducted in this area. Objective: To review and classify previous research, and to identify insights into the challenges research confronts when retrieving and mining practitioners' self-reports of their experience of software practice. Method: We conduct an exploratory review to identify and classify 42 articles. We analyse a selection of those articles for insights on challenges to mining professional experience. Results: We identify only one directly relevant article. Even then this article concerns the software professional's emotional experiences rather than the professional's reporting of behaviour and events occurring during software practice. We discuss challenges concerning: the prevalence of professional experience; definitions, models and theories; the sparseness of data; units of discourse analysis; annotator agreement; evaluation of the performance of algorithms; and the lack of replications. Conclusion: No directly relevant prior research appears to have been conducted in this area. We discuss the value of reporting negative results in secondary studies. There are a range of research opportunities but also considerable challenges. We formulate a set of guiding questions for further research in this area.

Vahid Garousi, Austen Rainer, Per Lauvås et al.

Context: With the rising complexity and scale of software systems, there is an ever-increasing demand for sophisticated and cost-effective software testing. To meet such a demand, there is a need for a highly-skilled software testing work-force (test engineers) in the industry. To address that need, many university educators worldwide have included software-testing education in their software engineering (SE) or computer science (CS) programs. Objective: Our objective in this paper is to summarize the body of experience and knowledge in the area of software-testing education to benefit the readers (both educators and researchers) in designing and delivering software testing courses in university settings, and to also conduct further education research in this area. Method: To address the above need, we conducted a systematic literature mapping (SLM) to synthesize what the community of educators have published on this topic. After compiling a candidate pool of 307 papers, and applying a set of inclusion/exclusion criteria, our final pool included 204 papers published between 1992 and 2019. Results: The topic of software-testing education is becoming more active, as we can see by the increasing number of papers. Many pedagogical approaches (how to best teach testing), course-ware, and specific tools for testing education have been proposed. Many challenges in testing education and insights on how to overcome those challenges have been proposed. Conclusion: This paper provides educators and researchers with a classification of existing studies within software-testing education. We further synthesize challenges and insights reported when teaching software testing. The paper also provides a reference ("index") to the vast body of knowledge and experience on teaching software testing.

Vahid Garousi, Michael Felderer, Mika V. Mäntylä et al.

Researchers generally place the most trust in peer-reviewed, published information, such as journals and conference papers. By contrast, software engineering (SE) practitioners typically do not have the time, access or expertise to review and benefit from such publications. As a result, practitioners are more likely to turn to other sources of information that they trust, e.g., trade magazines, online blog-posts, survey results or technical reports, collectively referred to as Grey Literature (GL). Furthermore, practitioners also share their ideas and experiences as GL, which can serve as a valuable data source for research. While GL itself is not a new topic in SE, using, benefitting and synthesizing knowledge from the GL in SE is a contemporary topic in empirical SE research and we are seeing that researchers are increasingly benefitting from the knowledge available within GL. The goal of this chapter is to provide an overview to GL in SE, together with insights on how SE researchers can effectively use and benefit from the knowledge and evidence available in the vast amount of GL.