testRNN: Coverage-guided Testing on Recurrent Neural Networks
This addresses the need for reliable testing in RNNs, particularly LSTMs used in sequential tasks like text processing, but it is incremental as it builds on existing testing concepts for neural networks.
The paper tackles the problem of verifying and validating long short-term memory networks (LSTMs) by introducing testRNN, the first coverage-guided testing tool for this class of RNNs, which empirically evaluates robustness using three novel LSTM structural test coverage metrics.
Recurrent neural networks (RNNs) have been widely applied to various sequential tasks such as text processing, video recognition, and molecular property prediction. We introduce the first coverage-guided testing tool, coined testRNN, for the verification and validation of a major class of RNNs, long short-term memory networks (LSTMs). The tool implements a generic mutation-based test case generation method, and it empirically evaluates the robustness of a network using three novel LSTM structural test coverage metrics. Moreover, it is able to help the model designer go through the internal data flow processing of the LSTM layer. The tool is available through: https://github.com/TrustAI/testRNN under the BSD 3-Clause licence.