Error-margin Analysis for Hidden Neuron Activation Labels
This work addresses a fundamental challenge in explainable AI for researchers and practitioners by proposing a novel perspective on neuron labeling, though it appears incremental as it builds on existing recall-focused methods.
The paper tackles the problem of understanding concept representation in neural networks by arguing that existing explainable AI methods focus only on recall (what activates a neuron) and neglect precision (neuron responses to other stimuli), introducing the concept of neuron labels error margin to address this gap.
Understanding how high-level concepts are represented within artificial neural networks is a fundamental challenge in the field of artificial intelligence. While existing literature in explainable AI emphasizes the importance of labeling neurons with concepts to understand their functioning, they mostly focus on identifying what stimulus activates a neuron in most cases, this corresponds to the notion of recall in information retrieval. We argue that this is only the first-part of a two-part job, it is imperative to also investigate neuron responses to other stimuli, i.e., their precision. We call this the neuron labels error margin.