Exposing Previously Undetectable Faults in Deep Neural Networks
This addresses the limitation of existing DNN testing methods for researchers and practitioners by enabling detection of faults that were previously undetectable, representing a significant advancement rather than an incremental improvement.
The paper tackles the problem of detecting previously undetectable faults in deep neural networks by introducing a novel testing method that generates fresh test inputs varying in high-level features, and demonstrates its ability to detect both injected and new faults in state-of-the-art DNNs where existing methods fail.
Existing methods for testing DNNs solve the oracle problem by constraining the raw features (e.g. image pixel values) to be within a small distance of a dataset example for which the desired DNN output is known. But this limits the kinds of faults these approaches are able to detect. In this paper, we introduce a novel DNN testing method that is able to find faults in DNNs that other methods cannot. The crux is that, by leveraging generative machine learning, we can generate fresh test inputs that vary in their high-level features (for images, these include object shape, location, texture, and colour). We demonstrate that our approach is capable of detecting deliberately injected faults as well as new faults in state-of-the-art DNNs, and that in both cases, existing methods are unable to find these faults.