When Cyber-Physical Systems Meet AI: A Benchmark, an Evaluation, and a Way Forward
This work addresses the problem of evaluating and improving AI-enabled cyber-physical systems for safety-critical domains like automotive and medical devices, though it is incremental as it builds on existing deep reinforcement learning methods.
The authors tackled the lack of benchmarks and understanding of AI-enabled cyber-physical systems by creating a public benchmark across seven domains and evaluating AI controllers against traditional ones, finding that AI does not always outperform traditional controllers, existing testing techniques are inadequate, and hybrid systems can achieve better performance.
Cyber-physical systems (CPS) have been broadly deployed in safety-critical domains, such as automotive systems, avionics, medical devices, etc. In recent years, Artificial Intelligence (AI) has been increasingly adopted to control CPS. Despite the popularity of AI-enabled CPS, few benchmarks are publicly available. There is also a lack of deep understanding on the performance and reliability of AI-enabled CPS across different industrial domains. To bridge this gap, we initiate to create a public benchmark of industry-level CPS in seven domains and build AI controllers for them via state-of-the-art deep reinforcement learning (DRL) methods. Based on that, we further perform a systematic evaluation of these AI-enabled systems with their traditional counterparts to identify the current challenges and explore future opportunities. Our key findings include (1) AI controllers do not always outperform traditional controllers, (2) existing CPS testing techniques (falsification, specifically) fall short of analyzing AI-enabled CPS, and (3) building a hybrid system that strategically combines and switches between AI controllers and traditional controllers can achieve better performance across different domains. Our results highlight the need for new testing techniques for AI-enabled CPS and the need for more investigations into hybrid CPS systems to achieve optimal performance and reliability.