Metamorphic Testing of a Deep Learning based Forecaster
This work addresses the challenge of ensuring reliability in deep learning forecasting systems for system monitoring, representing an incremental improvement in testing methodologies.
The paper tackled the problem of testing a deep learning-based forecasting application for system outages by developing 19 metamorphic relations, which uncovered 8 previously unknown issues in the actual application and detected 65.9% of hypothetical bugs in a reference implementation.
In this paper, we present the Metamorphic Testing of an in-use deep learning based forecasting application. The application looks at the past data of system characteristics (e.g. `memory allocation') to predict outages in the future. We focus on two statistical / machine learning based components - a) detection of co-relation between system characteristics and b) estimating the future value of a system characteristic using an LSTM (a deep learning architecture). In total, 19 Metamorphic Relations have been developed and we provide proofs & algorithms where applicable. We evaluated our method through two settings. In the first, we executed the relations on the actual application and uncovered 8 issues not known before. Second, we generated hypothetical bugs, through Mutation Testing, on a reference implementation of the LSTM based forecaster and found that 65.9% of the bugs were caught through the relations.