Understanding the Importance of Single Directions via Representative Substitution
This provides new insights for researchers in interpretability of DNNs, though it is incremental as it builds on prior work.
The paper tackles the counterintuitive finding that interpretable units in deep neural networks (DNNs) have poor contributions to generalization, by introducing Representative Substitution (RS) to show these units have high RS and are not critical to generalization.
Understanding the internal representations of deep neural networks (DNNs) is crucal to explain their behavior. The interpretation of individual units, which are neurons in MLPs or convolution kernels in convolutional networks, has been paid much attention given their fundamental role. However, recent research (Morcos et al. 2018) presented a counterintuitive phenomenon, which suggests that an individual unit with high class selectivity, called interpretable units, has poor contributions to generalization of DNNs. In this work, we provide a new perspective to understand this counterintuitive phenomenon, which makes sense when we introduce Representative Substitution (RS). Instead of individually selective units with classes, the RS refers to the independence of a unit's representations in the same layer without any annotation. Our experiments demonstrate that interpretable units have high RS which are not critical to network's generalization. The RS provides new insights into the interpretation of DNNs and suggests that we need to focus on the independence and relationship of the representations.