Eric Knauss

Hans-Martin Heyn, Eric Knauss, Patrizio Pelliccione

Artificial intelligence (AI) in its various forms finds more and more its way into complex distributed systems. For instance, it is used locally, as part of a sensor system, on the edge for low-latency high-performance inference, or in the cloud, e.g. for data mining. Modern complex systems, such as connected vehicles, are often part of an Internet of Things (IoT). To manage complexity, architectures are described with architecture frameworks, which are composed of a number of architectural views connected through correspondence rules. Despite some attempts, the definition of a mathematical foundation for architecture frameworks that are suitable for the development of distributed AI systems still requires investigation and study. In this paper, we propose to extend the state of the art on architecture framework by providing a mathematical model for system architectures, which is scalable and supports co-evolution of different aspects for example of an AI system. Based on Design Science Research, this study starts by identifying the challenges with architectural frameworks. Then, we derive from the identified challenges four rules and we formulate them by exploiting concepts from category theory. We show how compositional thinking can provide rules for the creation and management of architectural frameworks for complex systems, for example distributed systems with AI. The aim of the paper is not to provide viewpoints or architecture models specific to AI systems, but instead to provide guidelines based on a mathematical formulation on how a consistent framework can be built up with existing, or newly created, viewpoints. To put in practice and test the approach, the identified and formulated rules are applied to derive an architectural framework for the EU Horizon 2020 project ``Very efficient deep learning in the IoT" (VEDLIoT) in the form of a case study.

Hans-Martin Heyn, Khan Mohammad Habibullah, Eric Knauss et al.

Software that contains machine learning algorithms is an integral part of automotive perception, for example, in driving automation systems. The development of such software, specifically the training and validation of the machine learning components, require large annotated datasets. An industry of data and annotation services has emerged to serve the development of such data-intensive automotive software components. Wide-spread difficulties to specify data and annotation needs challenge collaborations between OEMs (Original Equipment Manufacturers) and their suppliers of software components, data, and annotations. This paper investigates the reasons for these difficulties for practitioners in the Swedish automotive industry to arrive at clear specifications for data and annotations. The results from an interview study show that a lack of effective metrics for data quality aspects, ambiguities in the way of working, unclear definitions of annotation quality, and deficits in the business ecosystems are causes for the difficulty in deriving the specifications. We provide a list of recommendations that can mitigate challenges when deriving specifications and we propose future research opportunities to overcome these challenges. Our work contributes towards the on-going research on accountability of machine learning as applied to complex software systems, especially for high-stake applications such as automated driving.

Hans-Martin Heyn, Eric Knauss, Iswarya Malleswaran et al.

Context and motivation: The development and operation of critical software that contains machine learning (ML) models requires diligence and established processes. Especially the training data used during the development of ML models have major influences on the later behaviour of the system. Runtime monitors are used to provide guarantees for that behaviour. Question / problem: We see major uncertainty in how to specify training data and runtime monitoring for critical ML models and by this specifying the final functionality of the system. In this interview-based study we investigate the underlying challenges for these difficulties. Principal ideas/results: Based on ten interviews with practitioners who develop ML models for critical applications in the automotive and telecommunication sector, we identified 17 underlying challenges in 6 challenge groups that relate to the challenge of specifying training data and runtime monitoring. Contribution: The article provides a list of the identified underlying challenges related to the difficulties practitioners experience when specifying training data and runtime monitoring for ML models. Furthermore, interconnection between the challenges were found and based on these connections recommendation proposed to overcome the root causes for the challenges.

Kevin Mika, René Griessl, Nils Kucza et al.

The VEDLIoT project aims to develop energy-efficient Deep Learning methodologies for distributed Artificial Intelligence of Things (AIoT) applications. During our project, we propose a holistic approach that focuses on optimizing algorithms while addressing safety and security challenges inherent to AIoT systems. The foundation of this approach lies in a modular and scalable cognitive IoT hardware platform, which leverages microserver technology to enable users to configure the hardware to meet the requirements of a diverse array of applications. Heterogeneous computing is used to boost performance and energy efficiency. In addition, the full spectrum of hardware accelerators is integrated, providing specialized ASICs as well as FPGAs for reconfigurable computing. The project's contributions span across trusted computing, remote attestation, and secure execution environments, with the ultimate goal of facilitating the design and deployment of robust and efficient AIoT systems. The overall architecture is validated on use-cases ranging from Smart Home to Automotive and Industrial IoT appliances. Ten additional use cases are integrated via an open call, broadening the range of application areas.

Hans-Martin Heyn, Padmini Subbiash, Jennifer Linder et al.

[Context and motivation] For automated driving systems, the operational context needs to be known in order to state guarantees on performance and safety. The operational design domain (ODD) is an abstraction of the operational context, and its definition is an integral part of the system development process. [Question / problem] There are still major uncertainties in how to clearly define and document the operational context in a diverse and distributed development environment such as the automotive industry. This case study investigates the challenges with context definitions for the development of perception functions that use machine learning for automated driving. [Principal ideas/results] Based on qualitative analysis of data from semi-structured interviews, the case study shows that there is a lack of standardisation for context definitions across the industry, ambiguities in the processes that lead to deriving the ODD, missing documentation of assumptions about the operational context, and a lack of involvement of function developers in the context definition. [Contribution] The results outline challenges experienced by an automotive supplier company when defining the operational context for systems using machine learning. Furthermore, the study collected ideas for potential solutions from the perspective of practitioners.

Marco Kuhrmann, Paolo Tell, Regina Hebig et al.

Together with many success stories, promises such as the increase in production speed and the improvement in stakeholders' collaboration have contributed to making agile a transformation in the software industry in which many companies want to take part. However, driven either by a natural and expected evolution or by contextual factors that challenge the adoption of agile methods as prescribed by their creator(s), software processes in practice mutate into hybrids over time. Are these still agile? In this article, we investigate the question: what makes a software development method agile? We present an empirical study grounded in a large-scale international survey that aims to identify software development methods and practices that improve or tame agility. Based on 556 data points, we analyze the perceived degree of agility in the implementation of standard project disciplines and its relation to used development methods and practices. Our findings suggest that only a small number of participants operate their projects in a purely traditional or agile manner (under 15%). That said, most project disciplines and most practices show a clear trend towards increasing degrees of agility. Compared to the methods used to develop software, the selection of practices has a stronger effect on the degree of agility of a given discipline. Finally, there are no methods or practices that explicitly guarantee or prevent agility. We conclude that agility cannot be defined solely at the process level. Additional factors need to be taken into account when trying to implement or improve agility in a software company. Finally, we discuss the field of software process-related research in the light of our findings and present a roadmap for future research.

Rashidah Kasauli, Eric Knauss, Joyce Nakatumba-Nabende et al.

[Context & motivation] Driven by the need for faster time-to-market and reduced development lead-time, large-scale systems engineering companies are adopting agile methods in their organizations. This agile transformation is challenging and it is common that adoption starts bottom-up with agile software teams within the context of traditional company structures. [Question/Problem] This creates the challenge of agile teams working within a document-centric and plan-driven (or waterfall) environment. While it may be desirable to take the best of both worlds, it is not clear how that can be achieved especially with respect to managing requirements in large-scale systems. [Principal ideas/Results] This paper presents an exploratory case study at an automotive company, focusing on two departments of a large-scale systems company that is in the process of company-wide agile adoption. [Contribution] We present challenges related to requirements engineering that agile teams face while working within a larger plan-driven context and propose potential strategies to mitigate the challenges. Challenges relate to, e.g., development teams not being aware of the high-level requirement and dealing with flexibility of writing user stories. We found that strategies for overcoming most of these challenges are still lacking and thus call for more research.

Salome Maro, Jan-Philipp Steghöfer, Eric Knauss et al.

Practitioners are poorly supported by the scientific literature when managing traceability information models (TIMs), which capture the structure and semantics of trace links. In practice, companies manage their TIMs in very different ways, even in cases where companies share many similarities. We present our findings from an in-depth focus group about TIM management with three different systems engineering companies. We find that the concrete needs of the companies as well as challenges such as scale and workflow integration are not considered by existing scientific work. We thus issue a call-to-arms for the requirements engineering and software and systems traceability communities, the two main communities for traceability research, to refocus their work on these practical problems.

Kai Stapel, Eric Knauss, Kurt Schneider et al.

Distributed software development is more difficult than co-located software development. One of the main reasons is that communication is more difficult in distributed settings. Defined processes and artifacts help, but cannot cover all information needs. Not communicating important project information, decisions and rationales can result in duplicate or extra work, delays or even project failure. Planning and managing a distributed project from an information flow perspective helps to facilitate available communication channels right from the start - beyond the documents and artifacts which are defined for a given development process. In this paper we propose FLOW Mapping, a systematic approach for planning and managing information flows in distributed projects. We demonstrate the feasibility of our approach with a case study in a distributed agile class room project. FLOW Mapping is sufficient to plan communication and to measure conformance to the communication strategy. We also discuss cost and impact of our approach.

Hans-Martin Heyn, Eric Knauss, Amna Pir Muhammad et al.

Availability of powerful computation and communication technology as well as advances in artificial intelligence enable a new generation of complex, AI-intense systems and applications. Such systems and applications promise exciting improvements on a societal level, yet they also bring with them new challenges for their development. In this paper we argue that significant challenges relate to defining and ensuring behaviour and quality attributes of such systems and applications. We specifically derive four challenge areas from relevant use cases of complex, AI-intense systems and applications related to industry, transportation, and home automation: understanding, determining, and specifying (i) contextual definitions and requirements, (ii) data attributes and requirements, (iii) performance definition and monitoring, and (iv) the impact of human factors on system acceptance and success. Solving these challenges will imply process support that integrates new requirements engineering methods into development approaches for complex, AI-intense systems and applications. We present these challenges in detail and propose a research roadmap.

Eric Knauss

Context: Software engineering researchers and practitioners rely on empirical evidence from the field. Thus, education of software engineers must include strong and applied education in empirical research methods. For most students, the master's thesis is the last, but also most applied form of this education in their studies. Problem: Especially thesis work in collaboration with industry requires that concerns of stakeholders from academia and practice are carefully balanced. It is possible, yet difficult to do high-impact empirical work within the timeframe of a typical thesis. In particular, if this research aims to provide practical value to industry, academic quality can suffer. Even though constructive research methods such as Design Science Research (DSR) exist, thesis projects repeatably struggle to apply them. Principle solution idea: DSR enables balancing such concerns by providing room both for knowledge questions and design work. Yet, only limited experience exists in our field on how to make this research method work within the context of a master's thesis. To enable running design science master's theses in collaboration with industry, we complement existing method descriptions and guidelines with our own experience and pragmatic advice to students, examiners, and supervisors in academia and industry. Method: This paper itself is based on DSR. Based on 12 design science theses over the last seven years, we collect common pitfalls and good practice from analysing the theses, the student-supervisor interaction, the supervisor-industry interaction, the examiner feedback, and, where available, reviewer comments on publications that are based on such theses. Results: We provide concrete advise for framing research questions, structuring a report, as well as for planning and conducting empirical work with practitioners.

Xavier Franch, Daniel Mendez, Andreas Vogelsang et al.

The relevance of Requirements Engineering (RE) research to practitioners is vital for a long-term dissemination of research results to everyday practice. Some authors have speculated about a mismatch between research and practice in the RE discipline. However, there is not much evidence to support or refute this perception. This paper presents the results of a study aimed at gathering evidence from practitioners about their perception of the relevance of RE research and at understanding the factors that influence that perception. We conducted a questionnaire-based survey of industry practitioners with expertise in RE. The participants rated the perceived relevance of 435 scientific papers presented at five top RE-related conferences. The 153 participants provided a total of 2,164 ratings. The practitioners rated RE research as essential or worthwhile in a majority of cases. However, the percentage of non-positive ratings is still higher than we would like. Among the factors that affect the perception of relevance are the research's links to industry, the research method used, and respondents' roles. The reasons for positive perceptions were primarily related to the relevance of the problem and the soundness of the solution, while the causes for negative perceptions were more varied. The respondents also provided suggestions for future research, including topics researchers have studied for decades, like elicitation or requirement quality criteria.

Rebekka Wohlrab, Jennifer Horkoff, Rashidah Kasauli et al.

Large-scale companies commonly face the challenge of managing relevant knowledge between different organizational groups, particularly in increasingly agile contexts. In previous studies, we found the importance of analyzing methodological islands (i.e., groups using different development methods than the surrounding organization) and boundary objects between them. In this paper, we propose a metamodel to better capture and analyze coordination and knowledge management in practice. Such a metamodel can allow practitioners to describe current practices, analyze issues, and design better-suited coordination mechanisms. We evaluated the conceptual model together with four large-scale companies developing complex systems. In particular, we derived an initial list of bad smells that can be leveraged to detect issues and devise suitable improvement strategies for inter-team coordination in large-scale development. We present the model, smells, and our evaluation results.

Rebekka Wohlrab, Patrizio Pelliccione, Ali Shahrokni et al.

Traceability is a key enabler of various activities in automotive software and systems engineering and required by several standards. However, most existing traceability management approaches do not consider that traceability is situated in constantly changing development contexts involving multiple stakeholders. Together with an automotive supplier, we analyzed how technology, business, and organizational factors raise the need for flexible traceability. We present how traceability can be evolved in the development lifecycle, from early elicitation of traceability needs to the implementation of mature traceability strategies. Moreover, we shed light on how traceability can be managed flexibly within an agile team and more formally when crossing team borders and organizational borders. Based on these insights, we present requirements for flexible tool solutions, supporting varying levels of data quality, change propagation, versioning, and organizational traceability.

Rashidah Kasauli, Rebekka Wohlrab, Eric Knauss et al.

Large-scale system development companies are increasingly adopting agile methods. While this adoption may improve lead-times, such companies need to balance two trade-offs: (i) the need to have a uniform, consistent development method on system level with the need for specialised methods for teams in different disciplines(e.g., hardware, software, mechanics, sales, support); (ii) the need for comprehensive documentation on system level with the need to have lightweight documentation enabling iterative and agile work. With specialised methods for teams, isolated teams work within larger ecosystems of plan-driven culture, i.e., teams become agile "islands". At the boundaries, these teams share knowledge which needs to be managed well for a correct system to be developed. While it is useful to support diverse and specialised methods, it is important to understand which islands are repeatedly encountered, the reasons or factors triggering their existence, and how best to handle coordination between them. Based on a multiple case study, this work presents a catalogue of islands and the boundary objects between them. We believe this work will be beneficial to practitioners aiming to understand their ecosystems and researchers addressing communication and coordination challenges in large-scale development.

Rebekka Wohlrab, Eric Knauss, Patrizio Pelliccione

In large-scale automotive companies, various requirements engineering (RE) practices are used across teams. RE practices manifest in Requirements Information Models (RIM) that define what concepts and information should be captured for requirements. Collaboration of practitioners from different parts of an organization is required to define a suitable RIM that balances support for diverse practices in individual teams with the alignment needed for a shared view and team support on system level. There exists no guidance for this challenging task. This paper presents a mixed methods study to examine the role of RIMs in balancing alignment and diversity of RE practices in four automotive companies. Our analysis is based on data from systems engineering tools, 11 semi-structured interviews, and a survey to validate findings and suggestions. We found that balancing alignment and diversity of RE practices is important to consider when defining RIMs. We further investigated enablers for this balance and actions that practitioners take to achieve it. From these factors, we derived and evaluated recommendations for managing RIMs in practice that take into account the lifecycle of requirements and allow for diverse practices across sub-disciplines in early development, while enforcing alignment of requirements that are close to release.

Jan-Philipp Steghöfer, Eric Knauss, Jennifer Horkoff et al.

Automotive companies increasingly adopt scaled agile methods to allow them to deal with their organisational and product complexity. Suitable methods are needed to ensure safety when developing automotive systems. On a small scale, R-Scrum and SafeScrum are two concrete suggestions for how to develop safety-critical systems using agile methods. However, for large-scale environments, existing frameworks like SAFe or LeSS do not support the development of safety-critical systems out of the box. We, therefore, aim to understand which challenges exist when developing safety-critical systems within large-scale agile industrial settings, in particular in the automotive domain. Based on an analysis of R-Scrum and SafeScrum, we conducted a focus group with three experts from industry to collect challenges in their daily work. We found challenges in the areas of living traceability, continuous compliance, and organisational flexibility. Among others, organisations struggle with defining a suitable traceability strategy, performing incremental safety analysis, and with integrating safety practices into their scaled way of working. Our results indicate a need to provide practical approaches to integrate safety work into large-scale agile development and point towards possible solutions, e.g., modular safety cases. Keywords: Scaled Agile, Safety-Critical Systems, Software Processes, R-Scrum, SafeScrum

Jennifer Horkoff, Juho Lindman, Imed Hammouda et al.

Traditionally, software APIs (application programming interfaces) have been viewed from a technical perspective, as a means to separate implementation from functional calls, and as a way to define a contract of software functionality. The technical benefits of APIs have been reported in numerous studies. Several reports from industry offer useful practical design considerations for APIs, including advice on collecting usage data, monetization strategies, and at what point to open an API to external parties. Although this advice can be useful, the focus is still often on the API, only without considering the role the API plays in the wider organization or how it fits into an organizational strategy. Our industrial experiences show that more information is needed about the challenges and best practices of API design and management in an organizational context. Furthermore, it has become apparent that APIs are able to play a key role as part of a strategic business plan for software-intensive companies. In this report, we present the results of our work with companies to build a framework that synthesizes and summarizes API strategies from several perspectives, including: strategic API lifecycle, lifecycle stage characteristics, lifecycle use cases and transition points, API layered architecture, BAPO, value modeling, ecosystem mapping with goal models, API metrics, goal models with metrics, and API governance guidance. Our research was carried out with an industry-academic collaboration. The framework was developed iteratively through workshops and discussions with company partners. We show how conceptual frameworks can be used to drive API strategy development, drawing on our research with several companies in practice.

Rebekka Wohlrab, Patrizio Pelliccione, Eric Knauss et al.

Agile methods are increasingly introduced in automotive companies in the attempt to become more efficient and flexible in the system development. The adoption of agile practices influences communication between stakeholders, but also makes companies rethink the management of artifacts and documentation like requirements, safety compliance documents, and architecture models. Practitioners aim to reduce irrelevant documentation, but face a lack of guidance to determine what artifacts are needed and how they should be managed. This paper presents artifacts, challenges, guidelines, and practices for the continuous management of systems engineering artifacts in automotive based on a theoretical and empirical understanding of the topic. In collaboration with 53 practitioners from six automotive companies, we conducted a design-science study involving interviews, a questionnaire, focus groups, and practical data analysis of a systems engineering tool. The guidelines suggest the distinction between artifacts that are shared among different actors in a company (boundary objects) and those that are used within a team (locally relevant artifacts). We propose an analysis approach to identify boundary objects and three practices to manage systems engineering artifacts in industry.

Rashidah Kasauli, Eric Knauss, Benjamin Kanagwa et al.

In the last decades, agile methods had a huge impact on how software is developed. In many cases, this has led to significant benefits, such as quality and speed of software deliveries to customers. However, safety-critical systems have widely been dismissed from benefiting from agile methods. Products that include safety critical aspects are therefore faced with a situation in which the development of safety-critical parts can significantly limit the potential speed-up through agile methods, for the full product, but also in the non-safety critical parts. For such products, the ability to develop safety-critical software in an agile way will generate a competitive advantage. In order to enable future research in this important area, we present in this paper a mapping of the current state of practice based on {a mixed method approach}. Starting from a workshop with experts from six large Swedish product development companies we develop a lens for our analysis. We then present a systematic mapping study on safety-critical systems and agile development through this lens in order to map potential benefits, challenges, and solution candidates for guiding future research.

Eric Knauss, Grischa Liebel, Jennifer Horkoff et al.

T-Reqs is a text-based requirements management solution based on the git version control system. It combines useful conventions, templates and helper scripts with powerful existing solutions from the git ecosystem and provides a working solution to address some known requirements engineering challenges in large-scale agile system development. Specifically, it allows agile cross-functional teams to be aware of requirements at system level and enables them to efficiently propose updates to those requirements. Based on our experience with T-Reqs, we i) relate known requirements challenges of large-scale agile system development to tool support; ii) list key requirements for tooling in such a context; and iii) propose concrete solutions for challenges.

Christian Bartelt, Manfred Broy, Christoph Herrmann et al.

Global software engineering has become a fact in many companies due to real necessity in practice. In contrast to co-located projects global projects face a number of additional software engineering challenges. Among them quality management has become much more difficult and schedule and budget overruns can be observed more often. Compared to co-located projects global software engineering is even more challenging due to the need for integration of different cultures, different languages, and different time zones - across companies, and across countries. The diversity of development locations on several levels seriously endangers an effective and goal-oriented progress of projects. In this position paper we discuss reasons for global development, sketch settings for distribution and views of orchestration of dislocated companies in a global project that can be seen as a "virtual project environment". We also present a collection of questions, which we consider relevant for global software engineering. The questions motivate further discussion to derive a research agenda in global software engineering.

Olga Liskin, Christoph Herrmann, Eric Knauss et al.

During acceptance testing customers assess whether a system meets their expectations and often identify issues that should be improved. These findings have to be communicated to the developers a task we observed to be error prone, especially in distributed teams. Here, it is normally not possible to have developer representatives from every site attend the test. Developers who were not present might misunderstand insufficiently documented findings. This hinders fixing the issues and endangers customer satisfaction. Integrated feedback systems promise to mitigate this problem. They allow to easily capture findings and their context. Correctly applied, this technique could improve feedback, while reducing customer effort. This paper collects our experiences from comparing acceptance testing with and without feedback systems in a distributed project. Our results indicate that this technique can improve acceptance testing if certain requirements are met. We identify key requirements feedback systems should meet to support acceptance testing.