Unit Impulse Response as an Explainer of Redundancy in a Deep Convolutional Neural Network
This work addresses redundancy in CNNs for researchers and practitioners, but it is incremental as it builds on existing methods for quantifying redundancy.
The paper tackles the problem of redundancy in convolutional neural networks (CNNs) by proposing a method to identify systemic redundancy across network depth using unit impulse response, demonstrating its application to interpret redundancy in example networks.
Convolutional neural networks (CNN) are generally designed with a heuristic initialization of network architecture and trained for a certain task. This often leads to overparametrization after learning and induces redundancy in the information flow paths within the network. This robustness and reliability is at the increased cost of redundant computations. Several methods have been proposed which leverage metrics that quantify the redundancy in each layer. However, layer-wise evaluation in these methods disregards the long-range redundancy which exists across depth on account of the distributed nature of the features learned by the model. In this paper, we propose (i) a mechanism to empirically demonstrate the robustness in performance of a CNN on account of redundancy across its depth, (ii) a method to identify the systemic redundancy in response of a CNN across depth using the understanding of unit impulse response, we subsequently demonstrate use of these methods to interpret redundancy in few networks as example. These techniques provide better insights into the internal dynamics of a CNN