Mapping Knowledge Representations to Concepts: A Review and New Perspectives
This is an incremental review that addresses the problem of interpretability in neural networks for researchers and practitioners, focusing on clarifying explanation goals.
The paper reviews research on associating neural network internal representations with human-understandable concepts, proposing a taxonomy based on deductive nomological explanations and theories of causality to clarify expectations from explanations, while identifying an ambiguity in the goal of model explainability.
The success of neural networks builds to a large extent on their ability to create internal knowledge representations from real-world high-dimensional data, such as images, sound, or text. Approaches to extract and present these representations, in order to explain the neural network's decisions, is an active and multifaceted research field. To gain a deeper understanding of a central aspect of this field, we have performed a targeted review focusing on research that aims to associate internal representations with human understandable concepts. In doing this, we added a perspective on the existing research by using primarily deductive nomological explanations as a proposed taxonomy. We find this taxonomy and theories of causality, useful for understanding what can be expected, and not expected, from neural network explanations. The analysis additionally uncovers an ambiguity in the reviewed literature related to the goal of model explainability; is it understanding the ML model or, is it actionable explanations useful in the deployment domain?