Coloring the Black Box: Visualizing neural network behavior with a self-introspective model
This provides a visualization tool for researchers to analyze neural network structure and vulnerabilities, though it is incremental as it builds on existing autoencoding and visualization techniques.
The authors tackled the problem of understanding neural network behavior by proposing a self-introspective model that visualizes hidden activations, revealing patterns linked to data categories and errors in fringe areas, with illustrative implementations on MNIST and CIFAR-10 datasets.
The following work presents how autoencoding all the possible hidden activations of a network for a given problem can provide insight about its structure, behavior, and vulnerabilities. The method, termed self-introspection, can show that a trained model showcases similar activation patterns (albeit randomly distributed due to initialization) when shown data belonging to the same category, and classification errors occur in fringe areas where the activations are not as clearly defined, suggesting some form of random, slowly varying, implicit encoding occurring within deep networks, that can be observed with this representation. Additionally, obtaining a low-dimensional representation of all the activations allows for (1) real-time model evaluation in the context of a multiclass classification problem, (2) the rearrangement of all hidden layers by their relevance in obtaining a specific output, and (3) the obtainment of a framework where studying possible counter-measures to noise and adversarial attacks is possible. Self-introspection can show how damaged input data can modify the hidden activations, producing an erroneous response. A few illustrative are implemented for feedforward and convolutional models and the MNIST and CIFAR-10 datasets, showcasing its capabilities as a model evaluation framework.