Understanding Distributed Representations of Concepts in Deep Neural Networks without Supervision
This addresses the need for interpretability in deep learning models without relying on human supervision, offering a novel approach for researchers and practitioners, though it is incremental in building on existing concept discovery methods.
The paper tackles the problem of interpreting deep learning classifiers by proposing an unsupervised method to discover distributed representations of concepts through principal neuron selection, resulting in the identification of coherent concepts, unlabeled subclasses, and misclassification causes across various layers.
Understanding intermediate representations of the concepts learned by deep learning classifiers is indispensable for interpreting general model behaviors. Existing approaches to reveal learned concepts often rely on human supervision, such as pre-defined concept sets or segmentation processes. In this paper, we propose a novel unsupervised method for discovering distributed representations of concepts by selecting a principal subset of neurons. Our empirical findings demonstrate that instances with similar neuron activation states tend to share coherent concepts. Based on the observations, the proposed method selects principal neurons that construct an interpretable region, namely a Relaxed Decision Region (RDR), encompassing instances with coherent concepts in the feature space. It can be utilized to identify unlabeled subclasses within data and to detect the causes of misclassifications. Furthermore, the applicability of our method across various layers discloses distinct distributed representations over the layers, which provides deeper insights into the internal mechanisms of the deep learning model.