Markku Oivo

Muhammad Azeem Akbar, Matteo Esposito, Sami Hyrynsalmi et al.

In the era of 6G, developing and managing software requires cutting-edge software engineering (SE) theories and practices tailored for such complexity across a vast number of connected edge devices. Our project aims to lead the development of sustainable methods and energy-efficient orchestration models specifically for edge environments, enhancing architectural support driven by AI for contemporary edge-to-cloud continuum computing. This initiative seeks to position Finland at the forefront of the 6G landscape, focusing on sophisticated edge orchestration and robust software architectures to optimize the performance and scalability of edge networks. Collaborating with leading Finnish universities and companies, the project emphasizes deep industry-academia collaboration and international expertise to address critical challenges in edge orchestration and software architecture, aiming to drive significant advancements in software productivity and market impact.

Sanja Aaramaa, Sandun Dasanayake, Markku Oivo et al.

Requirements volatility is a major issue in software (SW) development, causing problems such as project delays and cost overruns. Even though there is a considerable amount of research related to requirement volatility, the majority of it is inclined toward project management aspects. The relationship between SW architecture design and requirements volatility has not been researched widely, even though changing requirements may for example lead to higher defect density during testing. An exploratory case study was conducted to study how requirements volatility affects SW architecture design. Fifteen semi-structured, thematic interviews were conducted in the case company, which provides the selection of software products for business customers and consumers. The research revealed the factors, such as requirements uncertainty and dynamic business environment, causing requirements volatility in the case company. The study identified the challenges that requirements volatility posed to SW architecture design, including scheduling and architectural technical debt. In addition, this study discusses means of mitigating the factors that cause requirements volatility and addressing the challenges posed by requirements volatility. SW architects are strongly influenced by requirement volatility. Thus understanding the factors causing requirements volatility as well as means to mitigate the challenges has high industrial relevance.

Alireza Haghighatkhah, Mika Mäntylä, Markku Oivo et al.

Two heuristics namely diversity-based (DBTP) and history-based test prioritization (HBTP) have been separately proposed in the literature. Yet, their combination has not been widely studied in continuous integration (CI) environments. The objective of this study is to catch regression faults earlier, allowing developers to integrate and verify their changes more frequently and continuously. To achieve this, we investigated six open-source projects, each of which included several builds over a large time period. Findings indicate that previous failure knowledge seems to have strong predictive power in CI environments and can be used to effectively prioritize tests. HBTP does not necessarily need to have large data, and its effectiveness improves to a certain degree with larger history interval. DBTP can be used effectively during the early stages, when no historical data is available, and also combined with HBTP to improve its effectiveness. Among the investigated techniques, we found that history-based diversity using NCD Multiset is superior in terms of effectiveness but comes with relatively higher overhead in terms of method execution time. Test prioritization in CI environments can be effectively performed with negligible investment using previous failure knowledge, and its effectiveness can be further improved by considering dissimilarities among the tests.

Alireza Haghighatkhah, Mika Mäntylä, Markku Oivo et al.

A classical heuristic in software testing is to reward diversity, which implies that a higher priority must be assigned to test cases that differ the most from those already prioritized. This approach is commonly known as similarity-based test prioritization (SBTP) and can be realized using a variety of techniques. The objective of our study is to investigate whether SBTP is more effective at finding defects than random permutation, as well as determine which SBTP implementations lead to better results. To achieve our objective, we implemented five different techniques from the literature and conducted an experiment using the defects4j dataset, which contains 395 real faults from six real-world open-source Java programs. Findings indicate that running the most dissimilar test cases early in the process is largely more effective than random permutation (Vargha-Delaney A [VDA]: 0.76-0.99 observed using normalized compression distance). No technique was found to be superior with respect to the effectiveness. Locality-sensitive hashing was, to a small extent, less effective than other SBTP techniques (VDA: 0.38 observed in comparison to normalized compression distance), but its speed largely outperformed the other techniques (i.e., it was approximately 5-111 times faster). Our results bring to mind the well-known adage, "don't put all your eggs in one basket". To effectively consume a limited testing budget, one should spread it evenly across different parts of the system by running the most dissimilar test cases early in the testing process.

Woubshet Behutiye, Pilar Rodríguez, Markku Oivo et al.

Context-Agile software development (ASD) promotes minimal documentation and often prioritizes functional requirements over quality requirements (QRs). The minimal documentation emphasis may be beneficial in reducing time-to-market for software. However, it can also be a concern, especially with QRs, since they are challenging to specify and document and are crucial for software success. Therefore, understanding how practitioners perceive the importance of QR documentation is valuable because it can provide insight into how they approach this task. It also helps in developing models and guidelines that support the documentation of QRs in ASD, which is a research gap. Objective: We aim to understand practitioners' perceptions of QR documentation and factors influencing this task to derive a model that supports optimal QR documentation in ASD. Method: We conducted a multiple case study involving 12 participants from three cases that apply ASD. Please refer to the document to read the full version of the abstract.

Adrian Santos, Sira Vegas, Oscar Dieste et al.

Context: Test-driven development (TDD) is an agile software development approach that has been widely claimed to improve software quality. However, the extent to which TDD improves quality appears to be largely dependent upon the characteristics of the study in which it is evaluated (e.g., the research method, participant type, programming environment, etc.). The particularities of each study make the aggregation of results untenable. Objectives: The goal of this paper is to: increase the accuracy and generalizability of the results achieved in isolated experiments on TDD, provide joint conclusions on the performance of TDD across different industrial and academic settings, and assess the extent to which the characteristics of the experiments affect the quality-related performance of TDD. Method: We conduct a family of 12 experiments on TDD in academia and industry. We aggregate their results by means of meta-analysis. We perform exploratory analyses to identify variables impacting the quality-related performance of TDD. Results: TDD novices achieve a slightly higher code quality with iterative test-last development (i.e., ITL, the reverse approach of TDD) than with TDD. The task being developed largely determines quality. The programming environment, the order in which TDD and ITL are applied, or the learning effects from one development approach to another do not appear to affect quality. The quality-related performance of professionals using TDD drops more than for students. We hypothesize that this may be due to their being more resistant to change and potentially less motivated than students. Conclusion: Previous studies seem to provide conflicting results on TDD performance (i.e., positive vs. negative, respectively). We hypothesize that these conflicting results may be due to different study durations, experiment participants being unfamiliar with the TDD process...

Adrian Santos, Sira Vegas, Markku Oivo et al.

Context: It has been argued that software engineering replications are useful for verifying the results of previous experiments. However, it has not yet been agreed how to check whether the results hold across replications. Besides, some authors suggest that replications that do not verify the results of previous experiments can be used to identify contextual variables causing the discrepancies. Objective: Study how to assess the (dis)similarity of the results of SE replications when they are compared to verify the results of previous experiments and understand how to identify whether contextual variables are influencing results. Method: We run simulations to learn how different ways of comparing replication results behave when verifying the results of previous experiments. We illustrate how to deal with context-induced changes. To do this, we analyze three groups of replications from our own research on test-driven development and testing techniques. Results: The direct comparison of p-values and effect sizes does not appear to be suitable for verifying the results of previous experiments and examining the variables possibly affecting the results in software engineering. Analytical methods such as meta-analysis should be used to assess the similarity of software engineering replication results and identify discrepancies in results. Conclusion: The results achieved in baseline experiments should no longer be regarded as a result that needs to be reproduced, but as a small piece of evidence within a larger picture that only emerges after assembling many small pieces to complete the puzzle.

Woubshet Behutiye, Pertti Seppanen, Pilar Rodriguez et al.

Context: Quality requirements (QRs) have a significant role in the success of software projects. In agile software development (ASD), where working software is valued over comprehensive documentation, QRs are often under-specified or not documented. Consequently, they may be handled improperly and result in degraded software quality and increased maintenance costs. Investigating the documentation of QRs in ASD, would provide evidence on existing practices, tools and aspects considered in ASD that other practitioners might utilize to improve documentation and management of QRs in ASD. Although there are some studies examining documentation in ASD, those that specifically investigate the documentation of QRs in depth are lacking. Method: we conducted a multiple case study by interviewing 15 practitioners of four ASD cases, to provide empirical evidence on documentation of QRs in ASD. We also run workshops with two of the cases, to identify important aspects that ASD practitioners consider when documenting QRs in requirements management repositories. Result and conclusions: ASD companies approach documentation of QRs to fit the needs of their context. They used tools, backlogs,iterative prototypes,... Please refer to the PDF document for the remaining part of the abstract and the rest of the paper

Adrian Santos, Sira Vegas, Markku Oivo et al.

Context: Researchers from different groups and institutions are collaborating on building groups of experiments by means of replication (i.e., conducting groups of replications). Disparate aggregation techniques are being applied to analyze groups of replications. The application of unsuitable techniques to aggregate replication results may undermine the potential of groups of replications to provide in-depth insights from experiment results. Objectives: Provide an analysis procedure with a set of embedded guidelines to aggregate software engineering (SE) replication results. Method: We compare the characteristics of groups of replications for SE and other mature experimental disciplines such as medicine and pharmacology. In view of their differences, the limitations with regard to the joint data analysis of groups of SE replications and the guidelines provided in mature experimental disciplines to analyze groups of replications, we build an analysis procedure with a set of embedded guidelines specifically tailored to the analysis of groups of SE replications. We apply the proposed analysis procedure to a representative group of SE replications to illustrate its use. Results: All the information contained within the raw data should be leveraged during the aggregation of replication results. The analysis procedure that we propose encourages the use of stratified individual participant data and aggregated data in tandem to analyze groups of SE replications. Conclusion: The aggregation techniques used to analyze groups of replications should be justified in research articles. This will increase the reliability and transparency of joint results. The proposed guidelines should ease this endeavor.

Woubshet Behutiye, Pertti Karhapää, Lidia Lopez et al.

Context:Quality requirements (QRs) describe the desired quality of software, and they play an important role in the success of software projects. In agile software development (ASD), QRs are often ill-defined and not well addressed due to the focus on quickly delivering functionality. Rapid software development (RSD) approaches (e.g., continuous delivery and continuous deployment), which shorten delivery times, are more prone to neglect QRs. Despite the significance of QRs in both ASD and RSD, there is limited synthesized knowledge on their management in those approaches. Objective:This study aims to synthesize state-of-the-art knowledge about QR management in ASD and RSD, focusing on three aspects: bibliometric, strategies, and challenges. Research method:Using a systematic mapping study with a snowballing search strategy, we identified and structured the literature on QR management in ASD and RSD. Check the PDF file to see the full abstract and document.

Ahmad Banijamali, Pooyan Jamshidi, Pasi Kuvaja et al.

Connecting vehicles to cloud platforms has enabled innovative business scenarios while raising new quality concerns, such as reliability and scalability, which must be addressed by research. Cloud-native architectures based on microservices are a recent approach to enable continuous delivery and to improve service reliability and scalability. We propose an approach for restructuring cloud platform architectures in the automotive domain into a microservices architecture. To this end, we adopted and implemented microservices patterns from literature to design the cloud-native automotive architecture and conducted a laboratory experiment to evaluate the reliability and scalability of microservices in the context of a real-world project in the automotive domain called Eclipse Kuksa. Findings indicated that the proposed architecture could handle the continuous software delivery over-the-air by sending automatic control messages to a vehicular setting. Different patterns enabled us to make changes or interrupt services without extending the impact to others. The results of this study provide evidences that microservices are a potential design solution when dealing with service failures and high payload on cloud-based services in the automotive domain.

Sandun Dasanayake, Sanja Aaramaa, Jouni Markkula et al.

Requirements volatility is a major issue in software development, causing problems such as higher defect density, project delays and cost overruns. Software architecture that guides the overall vision of software product, is one of the areas that is greatly affected by requirements volatility. Since critical architecture decisions are made based on the requirements at hand, changes in requirements can result signifiant changes in architecture. With the wide adoption of agile software development, software architectures are designed to accommodate possible future changes. However, the changes has to be carefully managed as unnecessary and excessive changes can bring negative consequences. An exploratory case study was conducted to study the impact of requirements volatility on software architecture. Fifteen semi-structured, thematic interviews were conducted in a European software company. The research revealed poor communication, information distortion, and external dependencies as the main factors that cause requirement volatility and inadequate architecture documentation, inability to trace design rationale, and increased complexity as the main implications of requirements volatility on software architecture. Insights from software teams' awareness of the requirement volatility, factors contribute to it, and possible ways to mitigate its implications will be utilized to improve the management of requirement volatility during software architecting process.

Vahid Garousi, Markus Borg, Markku Oivo

Software engineering (SE) research should be relevant to industrial practice. There have been regular discussions in the SE community on this issue since the 1980's, led by pioneers such as Robert Glass. As we recently passed the milestone of "50 years of software engineering", some recent positive efforts have been made in this direction, e.g., establishing "industrial" tracks in several SE conferences. However, many researchers and practitioners believe that we, as a community, are still struggling with research relevance and utility. The goal of this paper is to synthesize the evidence and experience-based opinions shared on this topic so far in the SE community, and to encourage the community to further reflect and act on the research relevance. For this purpose, we have conducted a Multi-vocal Literature Review (MLR) of 54 systematically-selected sources (papers and non peer-reviewed articles). Instead of relying on and considering the individual opinions on research relevance, mentioned in each of the sources, the MLR aims to synthesize and provide the "holistic" view on the topic. The highlights of our MLR findings are as follows. The top three root causes of low relevance, discussed in the community, are: (1) Researchers having simplistic views (or wrong assumptions) about SE in practice; (2) Lack of connection with industry; and (3) Wrong identification of research problems. The top three suggestions for improving research relevance are: (1) Using appropriate research approaches such as action-research; (2) Choosing relevant research problems; and (3) Collaborating with industry. By synthesizing all the discussions on this important topic so far, this paper aims to encourage further discussions and actions in the community to increase our collective efforts to improve the research relevance.

Adrian Santos, Jaroslav Spisak, Markku Oivo et al.

Test-Driven Development (TDD), an agile development approach that enforces the construction of software systems by means of successive micro-iterative testing coding cycles, has been widely claimed to increase external software quality. In view of this, some managers at Paf-a Nordic gaming entertainment company-were interested in knowing how would TDD perform at their premises. Eventually, if TDD outperformed their traditional way of coding (i.e., YW, short for Your Way), it would be possible to switch to TDD considering the empirical evidence achieved at the company level. We conduct an experiment at Paf to evaluate the performance of TDD, YW and the reverse approach of TDD (i.e., ITL, short for Iterative-Test Last) on external quality. TDD outperforms YW and ITL at Paf. Despite the encouraging results, we cannot recommend Paf to immediately adopt TDD as the difference in performance between YW and TDD is small. However, as TDD looks promising at Paf, we suggest to move some developers to TDD and to run a future experiment to compare the performance of TDD and YW. TDD slightly outperforms ITL in controlled experiments for TDD novices. However, more industrial experiments are still needed to evaluate the performance of TDD in real-life contexts.

Prabhat Ram, Pilar Rodriguez, Markku Oivo

Existing scientific literature highlights the importance of metrics in Agile Software Development (ASD). Still, empirical investigation into metrics in ASD is scarce, particularly in identifying the rationale and the operational challenges associated with metrics. Under the Q-Rapids project (Horizon 2020), we conducted a multiple case study at four Agile companies, using the Goal Question Metric (GQM) approach, to investigate the rationale explaining the choice of process metrics in ASD, and challenges faced in operationalizing them. Results reflect that companies are interested in assessing process aspects like velocity, testing performance, and estimation accuracy, and they prefer custom metrics for these assessments. Companies use metrics as a means to access and even capitalize on the data, erstwhile inaccessible due to technical or process constraints. However, development context of a company can hinder metrics operationalization, manifesting primarily as unavailability of the data required to measure metrics...(check the paper for full Abstract)

Adrian Santos, Markku Oivo, Natalia Juristo

Software Engineering (SE) experiments are traditionally analyzed with statistical tests (e.g., t-tests, ANOVAs, etc.) that assume equally spread data across treatments (i.e., the homogeneity of variances assumption). Differences across treatments' variances in SE are not seen as an opportunity to gain insights on technology performance, but instead, as a hindrance to analyze the data. We have studied the role of variance in mature experimental disciplines such as medicine. We illustrate the extent to which variance may inform on technology performance by means of simulation. We analyze a real-life industrial experiment on Test-Driven Development (TDD) where variance may impact technology desirability. Evaluating the performance of technologies just based on means-as traditionally done in SE-may be misleading. Technologies that make developers resemble more to each other (i.e., technologies with smaller variances) may be more suitable if the aim is minimizing the risk of adopting them in real practice.

Adrian Santos, Janne Jarvinen, Jari Partanen et al.

Test-Driven Development (TDD) has been claimed to increase external software quality. However, the extent to which TDD increases external quality has been seldom studied in industrial experiments. We conduct four industrial experiments in two different companies to evaluate the performance of TDD on external quality. We study whether the performance of TDD holds across premises within the same company and across companies. We identify participant-level characteristics impacting results. Iterative-Test Last (ITL), the reverse approach of TDD, outperforms TDD in three out of four premises. ITL outperforms TDD in both companies. The larger the experience with unit testing and testing tools, the larger the difference in performance between ITL and TDD (in favour of ITL). Technological environment (i.e., programming language and testing tool) seems not to impact results. Evaluating participant-level characteristics impacting results in industrial experiments may ease the understanding of the performance of TDD in realistic settings.

Stefan Wagner, Daniel Méndez Fernández, Michael Felderer et al.

Requirements Engineering (RE) has established itself as a software engineering discipline during the past decades. While researchers have been investigating the RE discipline with a plethora of empirical studies, attempts to systematically derive an empirically-based theory in context of the RE discipline have just recently been started. However, such a theory is needed if we are to define and motivate guidance in performing high quality RE research and practice. We aim at providing an empirical and valid foundation for a theory of RE, which helps software engineers establish effective and efficient RE processes. We designed a survey instrument and theory that has now been replicated in 10 countries world-wide. We evaluate the propositions of the theory with bootstrapped confidence intervals and derive potential explanations for the propositions. We report on the underlying theory and the full results obtained from the replication studies with participants from 228 organisations. Our results represent a substantial step forward towards developing an empirically-based theory of RE giving insights into current practices with RE processes. The results reveal, for example, that there are no strong differences between organisations in different countries and regions, that interviews, facilitated meetings and prototyping are the most used elicitation techniques, that requirements are often documented textually, that traces between requirements and code or design documents is common, requirements specifications themselves are rarely changed and that requirements engineering (process) improvement endeavours are mostly intrinsically motivated. Our study establishes a theory that can be used as starting point for many further studies for more detailed investigations. Practitioners can use the results as theory-supported guidance on selecting suitable RE methods and techniques.

Woubshet Behutiye, Pertti Karhapää, Dolors Costal et al.

Non-functional requirements (NFRs) are determinant for the success of software projects. However,they are characterized as hard to define, and in agile software development(ASD), are often given less priority and usually not documented. In this paper, we present the findings of the documentation practices and challenges of NFRs in companies utilizing ASD and propose guidelines for enhancing NFRs documentation in ASD. We interviewed practitioners from four companies and identified that epics, features, user stories, acceptance criteria,Definition of Done(DoD), product and sprint backlogs are used for documenting NFRS. Please refer to the manuscript for the full abstract.

Nebojša Taušan, Jouni Markkula, Pasi Kuvaja et al.

Software companies that develop their products on a basis of service-oriented architecture (SOA) can expect various improvements as a result of choreography. Current choreography practices, however, are not yet used extensively in the embedded systems domain even though SOA is increasingly used in this domain. The objective of this study is to identify current features of the use of choreography in the embedded systems domain for practitioners and researchers by systematically analysing current developments in the scientific literature, strategies for choreography adaption, choreography specification and execution types, and implicit assumptions about choreography. To fulfil this objective, a systematic literature review of scientific publications that focus on the use of choreography in the embedded systems domain was carried out. After screening, 48 publications were selected as primary studies and analysed using thematic synthesis. The main results of the study showed that there are differences in how choreography is used in embedded and non-embedded systems domain. In the embedded systems domain, it is used to capture the service interactions of a single organisation, while, for example, in the enterprise systems domain it captures the service interactions among multiple organisations. Additionally, the results indicate that the use of choreography can lead to improvements in system performance and that the languages that are used for choreography modelling in the embedded systems domain are insufficiently expressive to capture the complexities that are typical in this domain. The study results facilitate the work of practitioners by allowing them to make informed decisions about the applicability of choreography in their organisations.

Sandun Dasanayake, Jouni Markkula, Sanja Aaramaa et al.

Architecture decision making is considered one of the most challenging cognitive tasks in software development. The objective of this study is to explore the state of the practice of architecture decision making in software teams, including the role of the architect and the associated challenges. An exploratory case study was conducted in a large software company in Europe and fifteen software architects were interviewed as the primary method of data collection. The results reveal that the majority of software teams make architecture decisions collaboratively. Especially, the consultative decision- making style is preferred as it helps to make decisions efficiently while taking the opinions of the team members into consideration. It is observed that most of the software architects maintain a close relationship with the software teams. Several organisational, process and human related challenges and their impact on architecture decision-making are also identified.

Davide Fucci, Hakan Erdogmus, Burak Turhan et al.

Background: Test-driven development (TDD) is a technique that repeats short coding cycles interleaved with testing. The developer first writes a unit test for the desired functionality, followed by the necessary production code, and refactors the code. Many empirical studies neglect unique process characteristics related to TDD iterative nature. Aim: We formulate four process characteristic: sequencing, granularity, uniformity, and refactoring effort. We investigate how these characteristics impact quality and productivity in TDD and related variations. Method: We analyzed 82 data points collected from 39 professionals, each capturing the process used while performing a specific development task. We built regression models to assess the impact of process characteristics on quality and productivity. Quality was measured by functional correctness. Result: Quality and productivity improvements were primarily positively associated with the granularity and uniformity. Sequencing, the order in which test and production code are written, had no important influence. Refactoring effort was negatively associated with both outcomes. We explain the unexpected negative correlation with quality by possible prevalence of mixed refactoring. Conclusion: The claimed benefits of TDD may not be due to its distinctive test-first dynamic, but rather due to the fact that TDD-like processes encourage fine-grained, steady steps that improve focus and flow.

Sandun Dasanayake, Jouni Markkula, Sanja Aaramaa et al.

Software architecture decision-making is critical to the success of a software system as software architecture sets the structure of the system, determines its qualities, and has far-reaching consequences throughout the system life cycle. The complex nature of the software development context and the importance of the problem has led the research community to develop several techniques, tools, and processes to assist software architects in making better decisions. Despite these effort, the adoption of such systematic approaches appears to be quite limited in practice. In addition, the practitioners are also facing new challenges as different software development methods suggest different approaches for architecture design. In this paper, we study the current software architecture decision-making practices in the industry using a case study conducted among professional software architects in three different companies in Europe. As a result, we identified different software architecture decision-making practices followed by the software teams as well as their reasons for following them, the challenges associated with them, and the possible improvements from the software architects' point of view. Based on that, we recognized that improving software architecture knowledge management can address most of the identified challenges and would result in better software architecture decision-making.