Concept Probing: Where to Find Human-Defined Concepts (Extended Version)
This addresses the challenge of efficiently interpreting neural networks for researchers and practitioners, though it appears incremental as it builds on existing concept probing techniques.
The paper tackles the problem of identifying which layer in a neural network is best for probing human-defined concepts, proposing an automated method based on informativeness and regularity of representations, and validates it with empirical analysis across models and datasets.
Concept probing has recently gained popularity as a way for humans to peek into what is encoded within artificial neural networks. In concept probing, additional classifiers are trained to map the internal representations of a model into human-defined concepts of interest. However, the performance of these probes is highly dependent on the internal representations they probe from, making identifying the appropriate layer to probe an essential task. In this paper, we propose a method to automatically identify which layer's representations in a neural network model should be considered when probing for a given human-defined concept of interest, based on how informative and regular the representations are with respect to the concept. We validate our findings through an exhaustive empirical analysis over different neural network models and datasets.