The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks' Depth
This work addresses the environmental impact of AI by making large models more efficient for simpler tasks, though it is incremental as it builds on prior knowledge transfer methods.
The paper tackles the problem of reducing the computational burden of over-parametrized deep neural networks for simpler downstream tasks by proposing an entropy-based importance metric (EASIER) to reduce network depth, achieving efficiency gains as assessed on image classification setups.
While deep neural networks are highly effective at solving complex tasks, large pre-trained models are commonly employed even to solve consistently simpler downstream tasks, which do not necessarily require a large model's complexity. Motivated by the awareness of the ever-growing AI environmental impact, we propose an efficiency strategy that leverages prior knowledge transferred by large models. Simple but effective, we propose a method relying on an Entropy-bASed Importance mEtRic (EASIER) to reduce the depth of over-parametrized deep neural networks, which alleviates their computational burden. We assess the effectiveness of our method on traditional image classification setups. Our code is available at https://github.com/VGCQ/EASIER.