SETA: Statistical Fault Attribution for Compound AI Systems
This work addresses robustness and safety testing for complex, multi-network AI systems, such as those in autonomous applications, though it is incremental as it builds on existing testing techniques.
The paper tackles the problem of robustness testing for compound AI systems, which are challenging with existing single-network methods, by proposing a modular framework that enables component-wise analysis and error propagation reasoning, successfully demonstrating fine-grained robustness analysis in a real-world autonomous rail inspection system.
Modern AI systems increasingly comprise multiple interconnected neural networks to tackle complex inference tasks. Testing such systems for robustness and safety entails significant challenges. Current state-of-the-art robustness testing techniques, whether black-box or white-box, have been proposed and implemented for single-network models and do not scale well to multi-network pipelines. We propose a modular robustness testing framework that applies a given set of perturbations to test data. Our testing framework supports (1) a component-wise system analysis to isolate errors and (2) reasoning about error propagation across the neural network modules. The testing framework is architecture and modality agnostic and can be applied across domains. We apply the framework to a real-world autonomous rail inspection system composed of multiple deep networks and successfully demonstrate how our approach enables fine-grained robustness analysis beyond conventional end-to-end metrics.