Software and application patterns for explanation methods
This work addresses the need for better software tools to understand neural network predictions, which is crucial for sensitive domains like healthcare and autonomous driving, but it is incremental as it builds on existing explanation algorithms.
The paper tackles the challenge of making explanation methods for neural networks more accessible and practical by introducing software and application patterns, focusing on efficient coding within deep learning frameworks and embedding in downstream implementations.
Deep neural networks successfully pervaded many applications domains and are increasingly used in critical decision processes. Understanding their workings is desirable or even required to further foster their potential as well as to access sensitive domains like medical applications or autonomous driving. One key to this broader usage of explaining frameworks is the accessibility and understanding of respective software. In this work we introduce software and application patterns for explanation techniques that aim to explain individual predictions of neural networks. We discuss how to code well-known algorithms efficiently within deep learning software frameworks and describe how to embed algorithms in downstream implementations. Building on this we show how explanation methods can be used in applications to understand predictions for miss-classified samples, to compare algorithms or networks, and to examine the focus of networks. Furthermore, we review available open-source packages and discuss challenges posed by complex and evolving neural network structures to explanation algorithm development and implementations.