NEPENTHE: Entropy-Based Pruning as a Neural Network Depth's Reducer
This addresses the problem of high computational demands in deep neural networks for real-time and resource-limited applications, though it is incremental as it builds on existing pruning techniques.
The paper tackles the computational burden of deep neural networks by proposing NEPENTHE, an entropy-based pruning method that reduces network depth, showing it can linearize layers in models like MobileNet and Swin-T with little to no performance loss in over-parametrized regimes.
While deep neural networks are highly effective at solving complex tasks, their computational demands can hinder their usefulness in real-time applications and with limited-resources systems. Besides, for many tasks it is known that these models are over-parametrized: neoteric works have broadly focused on reducing the width of these networks, rather than their depth. In this paper, we aim to reduce the depth of over-parametrized deep neural networks: we propose an eNtropy-basEd Pruning as a nEural Network depTH's rEducer (NEPENTHE) to alleviate deep neural networks' computational burden. Based on our theoretical finding, NEPENTHE focuses on un-structurally pruning connections in layers with low entropy to remove them entirely. We validate our approach on popular architectures such as MobileNet and Swin-T, showing that when encountering an over-parametrization regime, it can effectively linearize some layers (hence reducing the model's depth) with little to no performance loss. The code will be publicly available upon acceptance of the article.