Runtime Monitoring Neuron Activation Patterns
This addresses safety concerns for neural network deployment in critical applications, but it is incremental as it builds on existing monitoring and activation pattern concepts.
The paper tackles the problem of ensuring neural network decisions in safety-critical domains are supported by training data by proposing runtime monitoring of neuron activation patterns, and experiments show that adjusting similarity thresholds can report a significant portion of misclassifications as unsupported with a small false-positive rate.
For using neural networks in safety critical domains, it is important to know if a decision made by a neural network is supported by prior similarities in training. We propose runtime neuron activation pattern monitoring - after the standard training process, one creates a monitor by feeding the training data to the network again in order to store the neuron activation patterns in abstract form. In operation, a classification decision over an input is further supplemented by examining if a pattern similar (measured by Hamming distance) to the generated pattern is contained in the monitor. If the monitor does not contain any pattern similar to the generated pattern, it raises a warning that the decision is not based on the training data. Our experiments show that, by adjusting the similarity-threshold for activation patterns, the monitors can report a significant portion of misclassfications to be not supported by training with a small false-positive rate, when evaluated on a test set.