Guilherme Horta Travassos

Guilherme H. Travassos, Sabrina Rocha, Rodrigo Feitosa et al.

The advances and availability of technologies involving Generative Artificial Intelligence (AI) are evolving clearly and explicitly, driving immediate changes in various work activities. Software Engineering (SE) is no exception and stands to benefit from these new technologies, enhancing productivity and quality in its software development processes. However, although the use of Generative AI in SE practices is still in its early stages, considering the lack of conclusive results from ongoing research and the limited technological maturity, we have chosen to incorporate these technologies in the development of a web-based software system to be used in clinical trials by a thoracic diseases research group at our university. For this reason, we decided to share this experience report documenting our development team's learning journey in using Generative AI during the software development process. Project management, requirements specification, design, development, and quality assurance activities form the scope of observation. Although we do not yet have definitive technological evidence to evolve our development process significantly, the results obtained and the suggestions shared here represent valuable insights for software organizations seeking to innovate their development practices to achieve software quality with generative AI.

Matteo Esposito, Mikel Robredo, Murali Sridharan et al.

Context: Empirical Software Engineering (ESE) drives innovation in SE through qualitative and quantitative studies. However, concerns about the correct application of empirical methodologies have existed since the 2006 Dagstuhl seminar on SE. Objective: To analyze three decades of SE research, identify mistakes in statistical methods, and evaluate experts' ability to detect and address these issues. Methods: We conducted a literature survey of ~27,000 empirical studies, using LLMs to classify statistical methodologies as adequate or inadequate. Additionally, we selected 30 primary studies and held a workshop with 33 ESE experts to assess their ability to identify and resolve statistical issues. Results: Significant statistical issues were found in the primary studies, and experts showed limited ability to detect and correct these methodological problems, raising concerns about the broader ESE community's proficiency in this area. Conclusions. Despite our study's eventual limitations, its results shed light on recurring issues from promoting information copy-and-paste from past authors' works and the continuous publication of inadequate approaches that promote dubious results and jeopardize the spread of the correct statistical strategies among researchers. Besides, it justifies further investigation into empirical rigor in software engineering to expose these recurring issues and establish a framework for reassessing our field's foundation of statistical methodology application. Therefore, this work calls for critically rethinking and reforming data analysis in empirical software engineering, paving the way for our work soon.

Santiago del Rey, Paulo Sérgio Medeiros dos Santos, Guilherme Horta Travassos et al.

Background: As empirical software engineering evolves, more studies adopt data strategies$-$approaches that investigate digital artifacts such as models, source code, or system logs rather than relying on human subjects. Synthesizing results from such studies introduces new methodological challenges. Aims: This study assesses the effects of model quantization on correctness and resource efficiency in deep learning (DL) systems. Additionally, it explores the methodological implications of aggregating evidence from empirical studies that adopt data strategies. Method: We conducted a research synthesis of six primary studies that empirically evaluate model quantization. We applied the Structured Synthesis Method (SSM) to aggregate the findings, which combines qualitative and quantitative evidence through diagrammatic modeling. A total of 19 evidence models were extracted and aggregated. Results: The aggregated evidence indicates that model quantization weakly negatively affects correctness metrics while consistently improving resource efficiency metrics, including storage size, inference latency, and GPU energy consumption$-$a manageable trade-off for many DL deployment contexts. Evidence across quantization techniques remains fragmented, underscoring the need for more focused empirical studies per technique. Conclusions: Model quantization offers substantial efficiency benefits with minor trade-offs in correctness, making it a suitable optimization strategy for resource-constrained environments. This study also demonstrates the feasibility of using SSM to synthesize findings from data strategy-based research.

Bruno Pedraca de Souza, Guilherme Horta Travassos

Software systems on the Internet of Things have driven the world into a new industrial revolution, bringing with it new features and concerns such as autonomy, continuous device connectivity, and interaction among systems, users, and things. Nevertheless, building these types of systems is still a problematic activity due to their specific features. Empirical studies show the lack of technologies to support the construction of IoT software systems, in which different software artifacts should be created to ensure their quality. Thus, software inspection has emerged as an alternative evidence-based method to support the quality assurance of artifacts produced during the software development cycle. However, there is no knowledge of inspection techniques applicable to IoT software systems. Therefore, this research presents SCENARIOTCHECK, a Checklist-based Reading Technique for the Verification of IoT Scenarios. The checklist has been evaluated with experimental studies. This research shows that the technique has good results regarding cost-efficiency, efficiency, and IoT software system development effectiveness.

Santiago Matalonga, Domenico Amalfitano, Andrea Doreste et al.

Context: Context-aware contemporary software systems (CACSS) are mainstream. Furthermore, they present challenges for current engineering practices. These challenges are distinctively present when testing CACSS, as the variation of context deepens the limitations of available software testing practices and technologies. Objective: To understand how the industry deals with the variation of context when testing CACSS. Method: A Rapid Review was commissioned to uncover the necessary evidence to achieve the objectives. Results: Our results show that current research initiatives aim to generate or improve Test Suites that can deal with the variation of context and the sheer volume of test input possibilities. To achieve this, they mostly rely on modelling the systems' dynamic behavior and increasing computing resources to generate test inputs. We found no evidence of research results aiming at managing context variation through the testing lifecycle process. Conclusions: We discuss how the identified solutions are not ready for mainstream adoption. They are all domain-specific, and while the ideas and approaches can be reproduced in different settings, the technologies noon to be re-engineered and tailor to the specific CACSS.

Danyllo Valente da Silva, Bruno Pedraça de Souza, Taisa Guidini Gonçalves et al.

Contemporary software systems (CSS), such as the internet of things (IoT) based software systems, incorporate new concerns and characteristics inherent to the network, software, hardware, context awareness, interoperability, and others, compared to conventional software systems. In this sense, requirements engineering (RE) plays a fundamental role in ensuring these software systems' correct development looking for the business and end-user needs. Several software technologies supporting RE are available in the literature, but many do not cover all CSS specificities, notably those based on IoT. This research article presents RETIoT (Requirements Engineering Technology for the Internet of Things based software systems), aiming to provide methodological, technical, and tooling support to produce IoT software system requirements document. It is composed of an IoT scenario description technique, a checklist to verify IoT scenarios, construction processes, and templates for IoT software systems. A feasibility study was carried out in IoT system projects to observe its templates and identify improvement opportunities. The results indicate the feasibility of RETIoT templates' when used to capture IoT characteristics. However, further experimental studies represent research opportunities, strengthen confidence in its elements (construction process, techniques, and templates), and capture end-user perception.

Rebeca Motta, Káthia Oliveira, Guilherme Travassos

The Roadmap is performed in the context of a Ph.D. research in collaboration between the Experimental Software Engineering Group, from the Systems Engineering and Computing Program of the Federal University of Rio de Janeiro (COPPE/UFRJ) and the Laboratory of Industrial and Human Automation Control, Mechanical engineering and Computer Science (LAMIH UMR CNRS 8201) in the Universitè Polytechnique Hauts-de-France (UPHF). The Roadmap resulted from an investigation on the particularities of IoT applications. It is the concrete organization of the concepts and evidence gathered from different experimental studies. It comes to support the definition of IoT software systems, with specific items for the project team to discuss and define the essential aspects related to the specifying, designing, and implementing an IoT application.

Victor Vidigal Ribeiro, Daniela Soares Cruzes, Guilherme Horta Travassos

Context: Security and performance (S&P) are critical non-functional requirements on software systems. Therefore, verification activities should be included in the development process to identify related defects and avoiding S&P failures after deployment. However, the state of the practice of S&P verification is unclear, challenging academia to offer solutions for real-world problems faced by the S&P verification practitioners. Thus, identifying factors moderating the S&P verification helps software development organizations improve the S&P verification, releasing software that meets security and performance requirements. Objective: To present moderator factors influencing S&P verification activities and actions to promote S&P moderator factors. Method: Multiple case study using qualitative analysis of observational data to identify S&P moderators factors. Literature Rapid Reviews with Snowballing to strengthen confidence in the identified S&P moderators factors. Practitioners Survey to classify the S&P moderator factors regarding their relevance. Results: Identification of eight S&P moderator factors regarding organizational awareness, crossfunctional team, S&P requirements, support tools, verification environment, verification methodology, verification planning, and reuse practices. The literature reviews allowed us to confirm the identified S&P moderator factors and identify a set of actions to promote each of them. A survey with 37 valid participants allowed us to classify the identified S&P moderators factors and their actions relevant to S&P verification activities. Conclusions: The S&P moderator factors can be considered key points in which software development organizations should invest to implement or improve S&P verification activities.

Rebeca Motta, Káthia de Oliveira, Guilherme Travassos

We conducted a set of Rapid Reviews to characterize Internet of Things facets. We formatted a generic meta-protocol that was instantiated for each of the six facets presented (Connectivity, Things, Behavior, Smartness, Interactivity, and Environment)and considering the issue of Security, one of the most important and frequent challenges in the context of IoT. The meta-protocol is detailed and the results of each review are presented.

Xavier Franch, Daniel Mendez, Andreas Vogelsang et al.

The relevance of Requirements Engineering (RE) research to practitioners is vital for a long-term dissemination of research results to everyday practice. Some authors have speculated about a mismatch between research and practice in the RE discipline. However, there is not much evidence to support or refute this perception. This paper presents the results of a study aimed at gathering evidence from practitioners about their perception of the relevance of RE research and at understanding the factors that influence that perception. We conducted a questionnaire-based survey of industry practitioners with expertise in RE. The participants rated the perceived relevance of 435 scientific papers presented at five top RE-related conferences. The 153 participants provided a total of 2,164 ratings. The practitioners rated RE research as essential or worthwhile in a majority of cases. However, the percentage of non-positive ratings is still higher than we would like. Among the factors that affect the perception of relevance are the research's links to industry, the research method used, and respondents' roles. The reasons for positive perceptions were primarily related to the relevance of the problem and the soundness of the solution, while the causes for negative perceptions were more varied. The respondents also provided suggestions for future research, including topics researchers have studied for decades, like elicitation or requirement quality criteria.

Michael Felderer, Guilherme Horta Travassos

Empirical methods like experimentation have become a powerful means to drive the field of software engineering by creating scientific evidence on software development, operation, and maintenance, but also by supporting practitioners in their decision making and learning. Today empirical methods are fully applied in software engineering. However, they have developed in several iterations since the 1960s. In this chapter we tell the history of empirical software engineering and present the evolution of empirical methods in software engineering in five iterations, i.e., (1) mid-1960s to mid-1970s, (2) mid-1970s to mid-1980s, (3) mid-1980s to end of the 1990s, (4) the 2000s, and (5) the 2010s. We present the five iterations of the development of empirical software engineering mainly from a methodological perspective and additionally take key papers, venues, and books, which are covered in chronological order in a separate section on recommended further readings, into account. We complement our presentation of the evolution of empirical software engineering by presenting the current situation and an outlook in Sect. 4 and the available books on empirical software engineering. Furthermore, based on the chapters covered in this book we discuss trends on contemporary empirical methods in software engineering related to the plurality of research methods, human factors, data collection and processing, aggregation and synthesis of evidence, and impact of software engineering research.

X. Franch, D. Méndez Fernández, M. Oriol et al.

The relevance of Requirements Engineering (RE) research to practitioners is a prerequisite for problem-driven research in the area and key for a long-term dissemination of research results to everyday practice. To better understand how industry practitioners perceive the practical relevance of RE research, we have initiated the RE-Pract project, an international collaboration conducting an empirical study. This project opts for a replication of previous work done in two different domains and relies on survey research. To this end, we have designed a survey to be sent to several hundred industry practitioners at various companies around the world and ask them to rate their perceived practical relevance of the research described in a sample of 418 RE papers published between 2010 and 2015 at the RE, ICSE, FSE, ESEC/FSE, ESEM and REFSQ conferences. In this paper, we summarise our research protocol and present the current status of our study and the planned future steps.

Marco Torchiano, Daniel Méndez Fernández, Guilherme Horta Travassos et al.

Context: Surveys constitute an valuable tool to capture a large-scale snapshot of the state of the practice. Apparently trivial to adopt, surveys hide, however, several pitfalls that might hinder rendering the result valid and, thus, useful. Goal: We aim at providing an overview of main pitfalls in software engineering surveys and report on practical ways to deal with them. Method: We build on the experiences we collected in conducting many studies and distill the main lessons learnt. Results: The eight lessons learnt we report cover different aspects of the survey process ranging from the design of initial research objectives to the design of a questionnaire. Conclusions: Our hope is that by sharing our lessons learnt, combined with a disciplined application of the general survey theory, we contribute to improving the quality of the research results achievable by employing software engineering surveys.