zoNNscan : a boundary-entropy index for zone inspection of neural models
This work addresses the problem of improving interpretability and safety for deep neural networks in critical systems, though it is incremental as it builds on existing entropy-based methods.
The authors tackled the problem of understanding decision boundary geometry in deep neural networks, which relates to issues like adversarial examples, by introducing zoNNscan, an index that measures boundary uncertainty around a given input point. They demonstrated that zoNNscan shows significantly higher values for adversarial examples and corner case inputs compared to standard inputs.
The training of deep neural network classifiers results in decision boundaries which geometry is still not well understood. This is in direct relation with classification problems such as so called adversarial examples. We introduce zoNNscan, an index that is intended to inform on the boundary uncertainty (in terms of the presence of other classes) around one given input datapoint. It is based on confidence entropy, and is implemented through sampling in the multidimensional ball surrounding that input. We detail the zoNNscan index, give an algorithm for approximating it, and finally illustrate its benefits on four applications, including two important problems for the adoption of deep networks in critical systems: adversarial examples and corner case inputs. We highlight that zoNNscan exhibits significantly higher values than for standard inputs in those two problem classes.