Robert M. Hierons

Andrew G. Clark, Neil Walkinshaw, Robert M. Hierons

Agent-based models play an important role in simulating complex emergent phenomena and supporting critical decisions. In this context, a software fault may result in poorly informed decisions that lead to disastrous consequences. The ability to rigorously test these models is therefore essential. In this systematic literature review, we answer five research questions related to the key aspects of test case generation in agent-based models: What are the information artifacts used to generate tests? How are these tests generated? How is a verdict assigned to a generated test? How is the adequacy of a generated test suite measured? What level of abstraction of an agent-based model is targeted by a generated test? Our results show that whilst the majority of techniques are effective for testing functional requirements at the agent and integration levels of abstraction, there are comparatively few techniques capable of testing society-level behaviour. Additionally, we identify a need for more thorough evaluation using realistic case studies that feature challenging properties associated with a typical agent-based model.

Zhe Chen, Yunyun Chen, Robert M. Hierons et al.

Runtime Verification (RV) is a lightweight formal technique in which program or system execution is monitored and analyzed, to check whether certain properties are satisfied or violated after a finite number of steps. The use of RV has led to interest in deciding whether a property is monitorable: whether it is always possible for the satisfaction or violation of the property to be determined after a finite future continuation. However, classical two-valued monitorability suffers from two inherent limitations. First, a property can only be evaluated as monitorable or non-monitorable; no information is available regarding whether only one verdict (satisfaction or violation) can be detected. Second, monitorability is defined at the language-level and does not tell us whether satisfaction or violation can be detected starting from the current monitor state during system execution. To address these limitations, this paper proposes a new notion of four-valued monitorability for $ω$-languages and applies it at the state-level. Four-valued monitorability is more informative than two-valued monitorability as a property can be evaluated as a four-valued result, denoting that only satisfaction, only violation, or both are active for a monitorable property. We can also compute state-level weak monitorability, i.e., whether satisfaction or violation can be detected starting from a given state in a monitor, which enables state-level optimizations of monitoring algorithms. Based on a new six-valued semantics, we propose procedures for computing four-valued monitorability of $ω$-regular languages, both at the language-level and at the state-level. We have developed a new tool that implements the proposed procedure for computing monitorability of LTL formulas.