HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks
This addresses the validation of DNNs in safety-critical applications like medical systems and autonomous cars, offering an incremental improvement over existing metamorphic testing methods.
The paper tackles the problem of validating deep neural networks (DNNs) for safety-critical systems by introducing HOMRS, an approach that automatically builds a small, optimized set of high-order metamorphic relations from elementary ones, showing it is more effective and computationally efficient than similar techniques like DeepXplore.
Deep Neural Networks (DNN) applications are increasingly becoming a part of our everyday life, from medical applications to autonomous cars. Traditional validation of DNN relies on accuracy measures, however, the existence of adversarial examples has highlighted the limitations of these accuracy measures, raising concerns especially when DNN are integrated into safety-critical systems. In this paper, we present HOMRS, an approach to boost metamorphic testing by automatically building a small optimized set of high order metamorphic relations from an initial set of elementary metamorphic relations. HOMRS' backbone is a multi-objective search; it exploits ideas drawn from traditional systems testing such as code coverage, test case, path diversity as well as input validation. We applied HOMRS to MNIST/LeNet and SVHN/VGG and we report evidence that it builds a small but effective set of high-order transformations that generalize well to the input data distribution. Moreover, comparing to similar generation technique such as DeepXplore, we show that our distribution-based approach is more effective, generating valid transformations from an uncertainty quantification point of view, while requiring less computation time by leveraging the generalization ability of the approach.