Context-Awareness and Interpretability of Rare Occurrences for Discovery and Formalization of Critical Failure Modes
This addresses safety risks in critical domains like surveillance and transportation by providing a scalable and interpretable method for failure detection, though it is incremental as it builds on existing ontology and human-in-the-loop approaches.
The paper tackles the problem of vulnerabilities in vision systems to rare scenarios in critical domains by introducing CAIRO, an ontology-based human-assistive framework for detecting and formalizing failure cases, resulting in test cases stored as knowledge graphs for sharing and analysis.
Vision systems are increasingly deployed in critical domains such as surveillance, law enforcement, and transportation. However, their vulnerabilities to rare or unforeseen scenarios pose significant safety risks. To address these challenges, we introduce Context-Awareness and Interpretability of Rare Occurrences (CAIRO), an ontology-based human-assistive discovery framework for failure cases (or CP - Critical Phenomena) detection and formalization. CAIRO by design incentivizes human-in-the-loop for testing and evaluation of criticality that arises from misdetections, adversarial attacks, and hallucinations in AI black-box models. Our robust analysis of object detection model(s) failures in automated driving systems (ADS) showcases scalable and interpretable ways of formalizing the observed gaps between camera perception and real-world contexts, resulting in test cases stored as explicit knowledge graphs (in OWL/XML format) amenable for sharing, downstream analysis, logical reasoning, and accountability.