On the Evolution of Neuron Communities in a Deep Learning Architecture
This work addresses the challenge of interpretability in deep learning for researchers and practitioners, offering incremental methods to analyze neuron behavior.
The paper tackled the problem of explaining deep learning model performance by analyzing neuron activation patterns, proposing graph-based community detection and entropy methods, and found that these approaches provide new insights into model performance.
Deep learning techniques are increasingly being adopted for classification tasks over the past decade, yet explaining how deep learning architectures can achieve state-of-the-art performance is still an elusive goal. While all the training information is embedded deeply in a trained model, we still do not understand much about its performance by only analyzing the model. This paper examines the neuron activation patterns of deep learning-based classification models and explores whether the models' performances can be explained through neurons' activation behavior. We propose two approaches: one that models neurons' activation behavior as a graph and examines whether the neurons form meaningful communities, and the other examines the predictability of neurons' behavior using entropy. Our comprehensive experimental study reveals that both the community quality and entropy can provide new insights into the deep learning models' performances, thus paves a novel way of explaining deep learning models directly from the neurons' activation pattern.