Normalized Relevance Measure as a Unifying Framework to Explain Neural Network Latent Structures
For researchers in explainable AI, this provides a mathematically grounded, unified framework for interpreting neural network latent structures, though its novelty is incremental as it subsumes existing methods.
The paper proposes the Normalized Relevance Measure (NRM) framework, a general explanation method that attributes relevance to arbitrary sets of neurons across layers of neural networks, unifying existing propagation-based approaches. It demonstrates utility in computer vision by revealing information flows in VGG16 networks.
To understand how a neural network (NN) functions and makes predictions, it has become increasingly clear that analyzing only the input domain is insufficient -- one must also examine its internal inference mechanisms to capture the complete picture. To explain the internal inference mechanisms of such models, it is essential to analyze the importance of latent representations for a given task. In this paper, we propose the \emph{normalized relevance measure} (NRM) framework -- a novel general explanation procedure that attributes relevance to \emph{arbitrary sets of neurons across layers of arbitrary architectures}. In the NRM framework, relevance of selected neurons is explicitly defined as a normalized signed measure, constructed using simple operations -- marginalization and conditioning based on additive and multiplicative laws -- in analogy to the probability measures. The normalization property further guarantees comparability across layers. The NRM framework subsumes existing propagation-based explanation algorithms by explicitly identifying the underlying quantity being computed. We demonstrate the utility of the framework in computer vision applications, where joint relevance analysis across multiple layers reveals key information flows in VGG16 networks. Overall, the NRM framework provides a general, mathematically grounded approach to understanding how modern NNs propagate information, offering a versatile and broadly applicable foundation for explainable artificial intelligence.