Importance-Driven Deep Learning System Testing
This work addresses the need for dependable testing in safety-critical deep learning applications, representing an incremental advancement over existing software engineering adaptations.
The paper tackles the problem of inadequate testing criteria for deep learning systems in safety-critical applications by introducing DeepImportance, a methodology with an Importance-Driven test adequacy criterion, which empirically demonstrates improved robustness across multiple datasets and adversarial techniques.
Deep Learning (DL) systems are key enablers for engineering intelligent applications due to their ability to solve complex tasks such as image recognition and machine translation. Nevertheless, using DL systems in safety- and security-critical applications requires to provide testing evidence for their dependable operation. Recent research in this direction focuses on adapting testing criteria from traditional software engineering as a means of increasing confidence for their correct behaviour. However, they are inadequate in capturing the intrinsic properties exhibited by these systems. We bridge this gap by introducing DeepImportance, a systematic testing methodology accompanied by an Importance-Driven (IDC) test adequacy criterion for DL systems. Applying IDC enables to establish a layer-wise functional understanding of the importance of DL system components and use this information to assess the semantic diversity of a test set. Our empirical evaluation on several DL systems, across multiple DL datasets and with state-of-the-art adversarial generation techniques demonstrates the usefulness and effectiveness of DeepImportance and its ability to support the engineering of more robust DL systems.