Layerwise Change of Knowledge in Neural Networks
This work provides insights into the learning behavior of DNNs, addressing a fundamental problem in interpretability for researchers in machine learning and AI, though it is incremental as it builds on existing interaction definitions.
The paper tackles the problem of understanding how deep neural networks (DNNs) evolve knowledge across layers by extending the definition of interactions to intermediate layers, quantifying and tracking newly emerged and forgotten interactions during forward propagation, which reveals changes in generalization capacity and feature representation instability.
This paper aims to explain how a deep neural network (DNN) gradually extracts new knowledge and forgets noisy features through layers in forward propagation. Up to now, although the definition of knowledge encoded by the DNN has not reached a consensus, Previous studies have derived a series of mathematical evidence to take interactions as symbolic primitive inference patterns encoded by a DNN. We extend the definition of interactions and, for the first time, extract interactions encoded by intermediate layers. We quantify and track the newly emerged interactions and the forgotten interactions in each layer during the forward propagation, which shed new light on the learning behavior of DNNs. The layer-wise change of interactions also reveals the change of the generalization capacity and instability of feature representations of a DNN.