Data Driven Testing of Cyber Physical Systems
This work addresses the problem of improving testing efficiency and reliability for developers and engineers working on consumer-grade cyber-physical systems, representing an incremental advancement over existing manual or simulation-based methods.
The paper tackles the challenge of generating fault-revealing test cases for cyber-physical systems by proposing an automated approach that uses data-driven models to identify scenarios where the system behaves unexpectedly, such as violating temperature constraints in a smart thermostat, resulting in the discovery of several pit-fails.
Consumer grade cyber-physical systems (CPS) are becoming an integral part of our life, automatizing and simplifying everyday tasks. Indeed, due to complex interactions between hardware, networking and software, developing and testing such systems is known to be a challenging task. Various quality assurance and testing strategies have been proposed. The most common approach for pre-deployment testing is to model the system and run simulations with models or software in the loop. In practice, most often, tests are run for a small number of simulations, which are selected based on the engineers' domain knowledge and experience. In this paper we propose an approach to automatically generate fault-revealing test cases for CPS. We have implemented our approach in Python, using standard frameworks and used it to generate scenarios violating temperature constraints for a smart thermostat implemented as a part of our IoT testbed. Data collected from an application managing a smart building have been used to learn models of the environment under ever changing conditions. The suggested approach allowed us to identify several pit-fails, scenarios (i.e., environment conditions and inputs), where the system behaves not as expected.