On-Off Pattern Encoding and Path-Count Encoding as Deep Neural Network Representations
This work addresses the challenge of interpreting DNN representations for researchers, but it is incremental as it builds on existing methods like CAM.
The paper tackled the problem of understanding encoded representations in Deep Neural Networks by analyzing On-Off patterns and PathCount in simple image classification tasks, showing that these representations can be used to improve the Class Activation Map method.
Understanding the encoded representation of Deep Neural Networks (DNNs) has been a fundamental yet challenging objective. In this work, we focus on two possible directions for analyzing representations of DNNs by studying simple image classification tasks. Specifically, we consider \textit{On-Off pattern} and \textit{PathCount} for investigating how information is stored in deep representations. On-off pattern of a neuron is decided as `on' or `off' depending on whether the neuron's activation after ReLU is non-zero or zero. PathCount is the number of paths that transmit non-zero energy from the input to a neuron. We investigate how neurons in the network encodes information by replacing each layer's activation with On-Off pattern or PathCount and evaluating its effect on classification performance. We also examine correlation between representation and PathCount. Finally, we show a possible way to improve an existing DNN interpretation method, Class Activation Map (CAM), by directly utilizing On-Off or PathCount.