Compact Neural Representation Using Attentive Network Pruning
This work addresses the challenge of deploying deep networks on resource-constrained devices, though it appears incremental as it builds on existing pruning methods.
The paper tackles the problem of reducing computational and memory demands of deep neural networks for mobile platforms by proposing an attentive network pruning method, achieving high compression ratios with negligible accuracy loss on benchmark datasets.
Deep neural networks have evolved to become power demanding and consequently difficult to apply to small-size mobile platforms. Network parameter reduction methods have been introduced to systematically deal with the computational and memory complexity of deep networks. We propose to examine the ability of attentive connection pruning to deal with redundancy reduction in neural networks as a contribution to the reduction of computational demand. In this work, we describe a Top-Down attention mechanism that is added to a Bottom-Up feedforward network to select important connections and subsequently prune redundant ones at all parametric layers. Our method not only introduces a novel hierarchical selection mechanism as the basis of pruning but also remains competitive with previous baseline methods in the experimental evaluation. We conduct experiments using different network architectures on popular benchmark datasets to show high compression ratio is achievable with negligible loss of accuracy.