Jessica Díaz

Jessica Díaz, Jorge Pérez, Carolina Gallardo et al.

In recent years, the qualitative research on empirical software engineering that applies Grounded Theory is increasing. Grounded Theory (GT) is a technique for developing theory inductively e iteratively from qualitative data based on theoretical sampling, coding, constant comparison, memoing, and saturation, as main characteristics. Large or controversial GT studies may involve multiple researchers in collaborative coding, which requires a kind of rigor and consensus that an individual coder does not. Although many qualitative researchers reject quantitative measures in favor of other qualitative criteria, many others are committed to measuring consensus through Inter-Rater Reliability (IRR) and/or Inter-Rater Agreement (IRA) techniques to develop a shared understanding of the phenomenon being studied. However, there are no specific guidelines about how and when to apply IRR/IRA during the iterative process of GT, so researchers have been using ad hoc methods for years. This paper presents a process for systematically applying IRR/IRA in GT studies that meets the iterative nature of this qualitative research method, which is supported by a previous systematic literature review on applying IRR/RA in GT studies in software engineering. This process allows researchers to incrementally generate a theory while ensuring consensus on the constructs that support it and, thus, improving the rigor of qualitative research. This formalization helps researchers to apply IRR/IRA to GT studies when various raters are involved in coding. Measuring consensus among raters promotes communicability, transparency, reflexivity, replicability, and trustworthiness of the research.

Daniel López-Fernández, Jessica Díaz, Javier García et al.

Context: DevOps can be defined as a cultural movement to improve and accelerate the delivery of business value by making the collaboration between development and operations effective. Objective: This paper aims to help practitioners and researchers to better understand the organizational structure and characteristics of teams adopting DevOps. Method: We conducted an exploratory study by leveraging in depth, semi-structured interviews to relevant stakeholders of 31 multinational software-intensive companies, together with industrial workshops and observations at organizations' facilities that supported triangulation. We used Grounded Theory as qualitative research method to explore the structure and characteristics of teams, and statistical analysis to discover their implications in software delivery performance. Results: We describe a taxonomy of team structure patterns that shows emerging, stable and consolidated product teams that are classified according to six variables, such as collaboration frequency, product ownership sharing, autonomy, among others, as well as their implications on software delivery performance. These teams are often supported by horizontal teams (DevOps platform teams, Centers of Excellence, and chapters) that provide them with platform technical capability, mentoring and evangelization, and even temporarily facilitate human resources. Conclusion: This study aims to strengthen evidence and support practitioners in making better informed about organizational team structures by analyzing their main characteristics and implications in software delivery performance.

Paul Ralph, Nauman bin Ali, Sebastian Baltes et al.

Empirical Standards are natural-language models of a scientific community's expectations for a specific kind of study (e.g. a questionnaire survey). The ACM SIGSOFT Paper and Peer Review Quality Initiative generated empirical standards for research methods commonly used in software engineering. These living documents, which should be continuously revised to reflect evolving consensus around research best practices, will improve research quality and make peer review more effective, reliable, transparent and fair.

Ángel González-Prieto, Jorge Perez, Jessica Diaz et al.

In recent years, the research on empirical software engineering that uses qualitative data analysis (e.g., cases studies, interview surveys, and grounded theory studies) is increasing. However, most of this research does not deep into the reliability and validity of findings, specifically in the reliability of coding in which these methodologies rely on, despite there exist a variety of statistical techniques known as Inter-Coder Agreement (ICA) for analyzing consensus in team coding. This paper aims to establish a novel theoretical framework that enables a methodological approach for conducting this validity analysis. This framework is based on a set of coefficients for measuring the degree of agreement that different coders achieve when judging a common matter. We analyze different reliability coefficients and provide detailed examples of calculation, with special attention to Krippendorff's $α$ coefficients. We systematically review several variants of Krippendorff's $α$ reported in the literature and provide a novel common mathematical framework in which all of them are unified through a universal $α$ coefficient. Finally, this paper provides a detailed guide of the use of this theoretical framework in a large case study on DevOps culture. We explain how $α$ coefficients are computed and interpreted using a widely used software tool for qualitative analysis like Atlas.ti. We expect that this work will help empirical researchers, particularly in software engineering, to improve the quality and trustworthiness of their studies.