Swaroopa Dola

Nusrat Jahan Mozumder, Felipe Toledo, Swaroopa Dola et al.

Testing allows developers to determine whether a system functions as expected. When such systems include deep neural networks (DNNs), Testing becomes challenging, as DNNs approximate functions for which the formalization of functional requirements is intractable. This prevents the application of well-developed approaches to requirements-based testing to DNNs. To address this, we propose a requirements-based testing method (RBT4DNN) that uses natural language requirements statements. These statements use a glossary of terms to define a semantic feature space that can be leveraged for test input generation. RBT4DNN formalizes preconditions of functional requirements as logical combinations of those semantic features. Training data matching these feature combinations can be used to fine-tune a generative model to reliably produce test inputs satisfying the precondition. Executing these tests on a trained DNN enables comparing its output to the expected requirement postcondition behavior. We propose two use cases for RBT4DNN: (1) given requirements defining DNN correctness properties, RBT4DNN comprises a novel approach for detecting faults, and (2) during development, requirements-guided exploration of model behavior can provide developers with feedback on model generalization. Our further evaluation shows that RBT4DNN-generated tests are realistic, diverse, and aligned with requirement preconditions, enabling targeted analysis of model behavior and effective fault detection.

Swaroopa Dola, Matthew B. Dwyer, Mary Lou Soffa

The reliability of software that has a Deep Neural Network (DNN) as a component is urgently important today given the increasing number of critical applications being deployed with DNNs. The need for reliability raises a need for rigorous testing of the safety and trustworthiness of these systems. In the last few years, there have been a number of research efforts focused on testing DNNs. However the test generation techniques proposed so far lack a check to determine whether the test inputs they are generating are valid, and thus invalid inputs are produced. To illustrate this situation, we explored three recent DNN testing techniques. Using deep generative model based input validation, we show that all the three techniques generate significant number of invalid test inputs. We further analyzed the test coverage achieved by the test inputs generated by the DNN testing techniques and showed how invalid test inputs can falsely inflate test coverage metrics. To overcome the inclusion of invalid inputs in testing, we propose a technique to incorporate the valid input space of the DNN model under test in the test generation process. Our technique uses a deep generative model-based algorithm to generate only valid inputs. Results of our empirical studies show that our technique is effective in eliminating invalid tests and boosting the number of valid test inputs generated.