Mario Gleirscher

Mario Gleirscher, Anne E. Haxthausen, Jan Peleska

In this paper, a quantitative risk assessment approach is discussed for the design of an obstacle detection function for low-speed freight trains with grade of automation (GoA)~4. In this 5-step approach, starting with single detection channels and ending with a three-out-of-three (3oo3) model constructed of three independent dual-channel modules and a voter, a probabilistic assessment is exemplified, using a combination of statistical methods and parametric stochastic model checking. It is illustrated that, under certain not unreasonable assumptions, the resulting hazard rate becomes acceptable for specific application settings. The statistical approach for assessing the residual risk of misclassifications in convolutional neural networks and conventional image processing software suggests that high confidence can be placed into the safety-critical obstacle detection function, even though its implementation involves realistic machine learning uncertainties.

Marie Farrell, Matt Luckcuck, Mario Gleirscher et al.

This EPTCS volume contains the proceedings for the Fifth International Workshop on Formal Methods for Autonomous Systems (FMAS 2023), which was held on the 15th and 16th of November 2023. FMAS 2023 was co-located with 18th International Conference on integrated Formal Methods (iFM) (iFM'22), organised by Leiden Institute of Advanced Computer Science of Leiden University. The workshop itself was held at Scheltema Leiden, a renovated 19th Century blanket factory alongside the canal. FMAS 2023 received 25 submissions. We received 11 regular papers, 3 experience reports, 6 research previews, and 5 vision papers. The researchers who submitted papers to FMAS 2023 were from institutions in: Australia, Canada, Colombia, France, Germany, Ireland, Italy, the Netherlands, Sweden, the United Kingdom, and the United States of America. Increasing our number of submissions for the third year in a row is an encouraging sign that FMAS has established itself as a reputable publication venue for research on the formal modelling and verification of autonomous systems. After each paper was reviewed by three members of our Programme Committee we accepted a total of 15 papers: 8 long papers and 7 short papers.

Mario Gleirscher, Stefan Kugele, Sven Linker

These are the proceedings of the Second International Workshop on Safe Control of Autonomous Vehicles, which took place on the 10th of April 2018 in Porto, Portugal as an affiliated workshop of CSPWeek. The task of this workshop is to identify open research problems, discuss recent achievements, bring together researchers in, e.g., control theory, adaptive systems, machine self-organization and autonomy, mobile intelligent robotics, transportation, traffic control, machine learning, software verification, and dependability and security engineering.

Jan Peleska, Felix Brüning, Mario Gleirscher et al.

This technical report presents research results achieved in the field of verification of trained Convolutional Neural Network (CNN) used for image classification in safety-critical applications. As running example, we use the obstacle detection function needed in future autonomous freight trains with Grade of Automation (GoA) 4. It is shown that systems like GoA 4 freight trains are indeed certifiable today with new standards like ANSI/UL 4600 and ISO 21448 used in addition to the long-existing standards EN 50128 and EN 50129. Moreover, we present a quantitative analysis of the system-level hazard rate to be expected from an obstacle detection function. It is shown that using sensor/perceptor fusion, the fused detection system can meet the tolerable hazard rate deemed to be acceptable for the safety integrity level to be applied (SIL-3). A mathematical analysis of CNN models is performed which results in the identification of classification clusters and equivalence classes partitioning the image input space of the CNN. These clusters and classes are used to introduce a novel statistical testing method for determining the residual error probability of a trained CNN and an associated upper confidence limit. We argue that this greybox approach to CNN verification, taking into account the CNN model's internal structure, is essential for justifying that the statistical tests have covered the trained CNN with its neurons and inter-layer mappings in a comprehensive way.

Till Schnittka, Mario Gleirscher

When designing correct-by-construction controllers for autonomous collectives, three key challenges are the task specification, the modelling, and its use at practical scale. In this paper, we focus on a simple yet useful abstraction for high-level controller synthesis for robot collectives with optimisation goals (e.g., maximum cleanliness, minimum energy consumption) and recurrence (e.g., re-establish contamination and charge thresholds) and safety (e.g., avoid full discharge, mutually exclusive room occupation) constraints. Due to technical limitations (related to scalability and using constraints in the synthesis), we simplify our graph-based setting from a stochastic two-player game into a single-player game on a partially observable Markov decision process (POMDP). Robustness against environmental uncertainty is encoded via partial observability. Linear-time correctness properties are verified separately after synthesising the POMDP strategy. We contribute at-scale guidance on POMDP modelling and controller synthesis for tasked robot collectives exemplified by the scenario of battery-driven robots responsible for cleaning public buildings with utilisation constraints.

Mario Gleirscher, Jaco van de Pol, Jim Woodcock

Formal methods were frequently shown to be effective and, perhaps because of that, practitioners are interested in using them more often. Still, these methods are far less applied than expected, particularly, in critical domains where they are strongly recommended and where they have the greatest potential. Our hypothesis is that formal methods still seem not to be applicable enough or ready for their intended use. In critical software engineering, what do we mean when we speak of a formal method? And what does it mean for such a method to be applicable both from a scientific and practical viewpoint? Based on what the literature tells about the first question, with this manifesto, we lay out a set of principles that when followed by a formal method give rise to its mature applicability in a given scope. Rather than exercising criticism of past developments, this manifesto strives to foster an increased use of formal methods to the maximum benefit.

Mario Gleirscher, Jaco van de Pol, Jim Woodcock

This volume contains the proceedings of the 1st International Workshop on Applicable Formal Methods (AppFM 2021), 23 November 2021, held online as part of the 24th International Symposium on Formal Methods (FM). The aim of the AppFM workshop is to bring together researchers who improve and evaluate existing formal approaches and new variants in practical contexts and support the transfer of these approaches to software engineering practice.

Mario Gleirscher, Jan Peleska

Verified controller synthesis uses world models that comprise all potential behaviours of humans, robots, further equipment, and the controller to be synthesised. A world model enables quantitative risk assessment, for example, by stochastic model checking. Such a model describes a range of controller behaviours some of which -- when implemented correctly -- guarantee that the overall risk in the actual world is acceptable, provided that the stochastic assumptions have been made to the safe side. Synthesis then selects an acceptable-risk controller behaviour. However, because of crossing abstraction, formalism, and tool boundaries, verified synthesis for robots and autonomous systems has to be accompanied by rigorous testing. In general, standards and regulations for safety-critical systems require testing as a key element to obtain certification credit before entry into service. This work-in-progress paper presents an approach to the complete testing of synthesised supervisory controllers that enforce safety properties in domains such as human-robot collaboration and autonomous driving. Controller code is generated from the selected controller behaviour. The code generator, however, is hard, if not infeasible, to verify in a formal and comprehensive way. Instead, utilising testing, an abstract test reference is generated, a symbolic finite state machine with simpler semantics than code semantics. From this reference, a complete test suite is derived and applied to demonstrate the observational equivalence between the synthesised abstract test reference and the generated concrete controller code running on a control system platform.

Mario Gleirscher, Radu Calinescu, James Douthwaite et al.

We present a tool-supported approach for the synthesis, verification and validation of the control software responsible for the safety of the human-robot interaction in manufacturing processes that use collaborative robots. In human-robot collaboration, software-based safety controllers are used to improve operational safety, e.g., by triggering shutdown mechanisms or emergency stops to avoid accidents. Complex robotic tasks and increasingly close human-robot interaction pose new challenges to controller developers and certification authorities. Key among these challenges is the need to assure the correctness of safety controllers under explicit (and preferably weak) assumptions. Our controller synthesis, verification and validation approach is informed by the process, risk analysis, and relevant safety regulations for the target application. Controllers are selected from a design space of feasible controllers according to a set of optimality criteria, are formally verified against correctness criteria, and are translated into executable code and validated in a digital twin. The resulting controller can detect the occurrence of hazards, move the process into a safe state, and, in certain circumstances, return the process to an operational state from which it can resume its original task. We show the effectiveness of our software engineering approach through a case study involving the development of a safety controller for a manufacturing work cell equipped with a collaborative robot.

Radu Calinescu, Naif Alasmari, Mario Gleirscher

We present a work-in-progress approach to improving driver attentiveness in cars provided with automated driving systems. The approach is based on a control loop that monitors the driver's biometrics (eye movement, heart rate, etc.) and the state of the car; analyses the driver's attentiveness level using a deep neural network; plans driver alerts and changes in the speed of the car using a formally verified controller; and executes this plan using actuators ranging from acoustic and visual to haptic devices. The paper presents (i) the self-adaptive system formed by this monitor-analyse-plan-execute (MAPE) control loop, the car and the monitored driver, and (ii) the use of probabilistic model checking to synthesise the controller for the planning step of the MAPE loop.

Mario Gleirscher

Safety controllers are system or software components responsible for handling risk in many machine applications. This tool paper describes a use case and a workflow for YAP, a research tool for risk modelling and discrete-event safety controller design. The goal of this use case is to derive a safety controller from hazard analysis and risk assessment, to define a design space for this controller, and to select a verified optimal controller instance from this design space. We represent this design space as a stochastic model and use YAP for risk modelling and generation of parts of this stochastic model. For the controller verification and selection step, we use a stochastic model checker. The approach is illustrated by an example of a collaborative robot operated in a manufacturing work cell.

Simon Foster, Yakoub Nemouchi, Mario Gleirscher et al.

Assurance cases are often required to certify critical systems. The use of formal methods in assurance can improve automation, increase confidence, and overcome errant reasoning. However, assurance cases can never be fully formalised, as the use of formal methods is contingent on models that are validated by informal processes. Consequently, assurance techniques should support both formal and informal artifacts, with explicated inferential links between them. In this paper, we contribute a formal machine-checked interactive language, called Isabelle/SACM, supporting the computer-assisted construction of assurance cases compliant with the OMG Structured Assurance Case Meta-Model. The use of Isabelle/SACM guarantees well-formedness, consistency, and traceability of assurance cases, and allows a tight integration of formal and informal evidence of various provenance. In particular, Isabelle brings a diverse range of automated verification techniques that can provide evidence. To validate our approach, we present a substantial case study based on the Tokeneer secure entry system benchmark. We embed its functional specification into Isabelle, verify its security requirements, and form a modular security case in Isabelle/SACM that combines the heterogeneous artifacts. We thus show that Isabelle is a suitable platform for critical systems assurance.

Mario Gleirscher, Nikita Johnson, Panayiotis Karachristou et al.

The coordinated assurance of interrelated critical properties, such as system safety and cyber-security, is one of the toughest challenges in critical systems engineering. In this chapter, we summarise approaches to the coordinated assurance of safety and security. Then, we highlight the state of the art and recent challenges in human-robot collaboration in manufacturing both from a safety and security perspective. We conclude with a list of procedural and technological issues to be tackled in the coordinated assurance of collaborative industrial robots.

Mario Gleirscher, Radu Calinescu

In human-robot collaboration (HRC), software-based automatic safety controllers (ASCs) are used in various forms (e.g. shutdown mechanisms, emergency brakes, interlocks) to improve operational safety. Complex robotic tasks and increasingly close human-robot interaction pose new challenges to ASC developers and certification authorities. Key among these challenges is the need to assure the correctness of ASCs under reasonably weak assumptions. To address this need, we introduce and evaluate a tool-supported ASC synthesis method for HRC in manufacturing. Our ASC synthesis is: (i) informed by the manufacturing process, risk analysis, and regulations; (ii) formally verified against correctness criteria; and (iii) selected from a design space of feasible controllers according to a set of optimality criteria. The synthesised ASC can detect the occurrence of hazards, move the process into a safe state, and, in certain circumstances, return the process to an operational state from which it can resume its original task.

Simon Foster, Mario Gleirscher, Radu Calinescu

The use of autonomous vehicles in real-world applications is often precluded by the difficulty of providing safety guarantees for their complex controllers. The simulation-based testing of these controllers cannot deliver sufficient safety guarantees, and the use of formal verification is very challenging due to the hybrid nature of the autonomous vehicles. Our work-in-progress paper introduces a formal verification approach that addresses this challenge by integrating the numerical computation of such a system (in GNU/Octave) with its hybrid system verification by means of a proof assistant (Isabelle). To show the effectiveness of our approach, we use it to verify differential invariants of an Autonomous Marine Vehicle with a controller switching between multiple modes.

Yakoub Nemouchi, Simon Foster, Mario Gleirscher et al.

Assurance cases are often required as a means to certify a critical system. Use of formal methods in assurance can improve automation, and overcome problems with ambiguity, faulty reasoning, and inadequate evidentiary support. However, assurance cases can rarely be fully formalised, as the use of formal methods is contingent on models validated by informal processes. Consequently, we need assurance techniques that support both formal and informal artifacts, with explicated inferential links and assumptions that can be checked by evaluation. Our contribution is a mechanical framework for developing assurance cases with integrated formal methods based in the Isabelle system. We demonstrate an embedding of the Structured Assurance Case Meta-model (SACM) using Isabelle/DOF, and show how this can be linked to formal analysis techniques originating from our verification framework, Isabelle/UTP. We validate our approach by mechanising a fragment of the Tokeneer security case, with evidence supplied by formal verification.

Mario Gleirscher

Inspired by widely-used techniques of causal modelling in risk, failure, and accident analysis, this work discusses a compositional framework for risk modelling. Risk models capture fragments of the space of risky events likely to occur when operating a machine in a given environment. Moreover, one can build such models into machines such as autonomous robots, to equip them with the ability of risk-aware perception, monitoring, decision making, and control. With the notion of a risk factor as the modelling primitive, the framework provides several means to construct and shape risk models. Relational and algebraic properties are investigated and proofs support the validity and consistency of these properties over the corresponding models. Several examples throughout the discussion illustrate the applicability of the concepts. Overall, this work focuses on the qualitative treatment of risk with the outlook of transferring these results to probabilistic refinements of the discussed framework.

Mario Gleirscher, Stefan Kugele

The specification, design, and assurance of safety encompasses various concepts and best practices, subject of reuse in form of patterns. This work summarizes applied research on such concepts and practices with a focus on the last two decades and on the state-of-the-art of patterns in safety-critical system design and assurance argumentation. We investigate several aspects of such patterns, for example, where and when they are applied, their characteristics and purposes, and how they are related. For each aspect, we provide an overview of relevant studies and synthesize a taxonomy of first principles underlying these patterns. Furthermore, we comment on how these studies address known challenges and we discuss suggestions for further research. Our findings disclose a lack of research on how patterns improve system safety claims and, vice versa, on the decomposition of system safety into separated local concerns, and on the impact of security on safety.

Mario Gleirscher, Simon Foster, Jim Woodcock

Formal methods have provided approaches for investigating software engineering fundamentals and also have high potential to improve current practices in dependability assurance. In this article, we summarise known strengths and weaknesses of formal methods. From the perspective of the assurance of robots and autonomous systems (RAS), we highlight new opportunities for integrated formal methods and identify threats to the adoption of such methods. Based on these opportunities and threats, we develop an agenda for fundamental and empirical research on integrated formal methods and for successful transfer of validated research to RAS assurance. Furthermore, we outline our expectations on useful outcomes of such an agenda.

Mario Gleirscher, Diego Marmsoler

Context: Formal methods (FMs) have been around for a while, still being unclear how to leverage their benefits, overcome their challenges, and set new directions for their improvement towards a more successful transfer into practice. Objective: We study the use of formal methods in mission-critical software domains, examining industrial and academic views. Method: We perform a cross-sectional on-line survey. Results: Our results indicate an increased intent to apply FMs in industry, suggesting a positively perceived usefulness. But the results also indicate a negatively perceived ease of use. Scalability, skills, and education seem to be among the key challenges to support this intent. Conclusions: We present the largest study of this kind so far (N = 216), and our observations provide valuable insights, highlighting directions for future theoretical and empirical research of formal methods. Our findings are strongly coherent with earlier observations by Austin and Parkin (1993).

Mario Gleirscher, Anne Nyokabi

System safety refers to a diverse engineering discipline assessing and improving various aspects of safety in socio-technical systems and their software-intensive sub-systems. While system safety has been a vital area of applied research for many decades, its practice and practitioners seem empirically still not well studied. Beyond mainly anecdotal evidence (interviews, on-line discussions), incident reports, and surveys, we are missing open, large-scale, and long-term investigations that promote knowledge transfer and research validation. We explore means for work safety practitioners rely on, factors influencing their performance, and their perception of their role in the system life cycle. Along with that we examine observations from previous research. We build a construct of safety practice, collect data for this construct using an on-line survey, summarise and interpret the collected data, and investigate several hypotheses based on the previous observations. We analyse and present the responses of 124 practitioners in safety-critical system and software projects. Aside from other findings, our data suggests that safety decision making mainly depends on expert opinion and project memory, lacks evidence that safety is typically a cost-benefit question, does not exhibit the prejudice that formal methods are not beneficial, leaves it unclear as to whether or not standards and methods have become inadequate, and indicates that safety is not typically confused with reliability. Additionally, we contribute a research design directing towards explanatory empirical studies of safety practice. Empirical research of safety practice is still in an early stage, bearing the risk of undesirable mismatches of the state of the art and the state of practice. However, this situation offers great opportunities for research.

Mario Gleirscher, Sven Linker, Stefan Kugele

This report summarizes the discussions, open issues, take-away messages, and conclusions of the 2nd SCAV workshop.

Mario Gleirscher, Stefan Kugele

Vehicle safety depends on (a) the range of identified hazards and (b) the operational situations for which mitigations of these hazards are acceptably decreasing risk. Moreover, with an increasing degree of autonomy, risk ownership is likely to increase for vendors towards regulatory certification. Hence, highly automated vehicles have to be equipped with verified controllers capable of reliably identifying and mitigating hazards in all possible operational situations. To this end, available methods for the design and verification of automated vehicle controllers have to be supported by models for hazard analysis and mitigation. In this paper, we describe (1) a framework for the analysis and design of planners (i.e., high-level controllers) capable of run-time hazard identification and mitigation, (2) an incremental algorithm for constructing planning models from hazard analysis, and (3) an exemplary application to the design of a fail-operational controller based on a given control system architecture. Our approach equips the safety engineer with concepts and steps to (2a) elaborate scenarios of endangerment and (2b) design operational strategies for mitigating such scenarios.

Mario Gleirscher

We assume that autonomous or highly automated driving (AD) will be accompanied by tough assurance obligations exceeding the requirements of even recent revisions of ISO 26262 or SOTIF. Hence, automotive control and safety engineers have to (i) comprehensively analyze the driving process and its control loop, (ii) identify relevant hazards stemming from this loop, (iii) establish feasible automated measures for the effective mitigation of these hazards or the alleviation of their consequences. By studying an example, this article investigates some achievements in the modeling for the steps (i), (ii), and (iii), amenable to formal verification of desired properties derived from potential assurance obligations such as the global existence of an effective mitigation strategy. In addition, the proposed approach is meant for step-wise refinement towards the automated synthesis of AD safety controllers implementing such properties.

Mario Gleirscher, Carmen Carlan

We observed that safety arguments are prone to stay too abstract, e.g. solutions refer to large packages, argument strategies to complex reasoning steps, contexts and assumptions lack traceability. These issues can reduce the confidence we require of such arguments. In this paper, we investigate the construction of confident arguments from (i) hazard analysis (HA) results and (ii) the design of safety measures, i.e., both used for confidence evaluation. We present an argument pattern integrating three HA techniques, i.e., FTA, FMEA, and STPA, as well as the reactions on the results of these analyses, i.e., safety requirements and design increments. We provide an example of how our pattern can help in argument construction and discuss steps towards using our pattern in formal analysis and computer-assisted construction of safety cases.

Mario Gleirscher, Dmitriy Golubitskiy, Maximilian Irlbeck et al.

Today, small and medium-sized enterprises (SMEs) in the software industry face major challenges. Their resource constraints require high efficiency in development. Furthermore, quality assurance (QA) measures need to be taken to mitigate the risk of additional, expensive effort for bug fixes or compensations. Automated static analysis (ASA) can reduce this risk because it promises low application effort. SMEs seem to take little advantage of this opportunity. Instead, they still mainly rely on the dynamic analysis approach of software testing. In this article, we report on our experiences from a technology transfer project. Our aim was to evaluate the results static analysis can provide for SMEs as well as the problems that occur when introducing and using static analysis in SMEs. We analysed five software projects from five collaborating SMEs using three different ASA techniques: code clone detection, bug pattern detection and architecture conformance analysis. Following the analysis, we applied a quality model to aggregate and evaluate the results. Our study shows that the effort required to introduce ASA techniques in SMEs is small (mostly below one person-hour each). Furthermore, we encountered only few technical problems. By means of the analyses, we could detect multiple defects in production code. The participating companies perceived the analysis results to be a helpful addition to their current QA and will include the analyses in their QA process. With the help of the Quamoco quality model, we could efficiently aggregate and rate static analysis results. However, we also encountered a partial mismatch with the opinions of the SMEs. We conclude, that ASA and quality models can be a valuable and affordable addition to the QA process of SMEs.

Mario Gleirscher, Dmitriy Golubitskiy, Maximilian Irlbeck et al.

Today's small and medium-sized enterprises (SMEs) in the software industry are faced with major challenges. While having to work efficiently using limited resources they have to perform quality assurance on their code to avoid the risk of further effort for bug fixes or compensations. Automated static analysis can reduce this risk because it promises little effort for running an analysis. We report on our experience in analysing five projects from and with SMEs by three different static analysis techniques: code clone detection, bug pattern detection and architecture conformance analysis. We found that the effort that was needed to introduce those techniques was small (mostly below one person-hour), that we can detect diverse defects in production code and that the participating companies perceived the usefulness of the presented techniques as well as our analysis results high enough to include the techniques in their quality assurance.

Sonila Dobi, Mario Gleirscher, Maria Spichkova et al.

Hazard and impact analysis is an indispensable task during the specification and development of safety-critical technical systems, and particularly of their software-intensive control parts. There is a lack of methods supporting an effective (reusable, automated) and integrated (cross-disciplinary) way to carry out such analyses. This report was motivated by an industrial project whose goal was to survey and propose methods and models for documentation and analysis of a system and its environment to support hazard and impact analysis as an important task of safety engineering and system development. We present and investigate three perspectives of how to properly encode safety-relevant domain knowledge for better reuse and automation, identify and assess all relevant hazards, as well as pre-process this information and make it easily accessible for reuse in other safety and systems engineering activities and, moreover, in similar engineering projects.