Layer Pruning with Consensus: A Triple-Win Solution
This work addresses the need for efficient and robust neural network pruning to reduce computational costs and environmental impact, representing an incremental improvement over existing layer-pruning methods.
The paper tackled the problem of layer pruning in neural networks by proposing a Consensus criterion that combines multiple similarity metrics to identify low-importance layers, resulting in up to 78.80% FLOPs reduction, performance on par with state-of-the-art methods, and improved robustness by up to 4 percentage points under adversarial attacks.
Layer pruning offers a promising alternative to standard structured pruning, effectively reducing computational costs, latency, and memory footprint. While notable layer-pruning approaches aim to detect unimportant layers for removal, they often rely on single criteria that may not fully capture the complex, underlying properties of layers. We propose a novel approach that combines multiple similarity metrics into a single expressive measure of low-importance layers, called the Consensus criterion. Our technique delivers a triple-win solution: low accuracy drop, high-performance improvement, and increased robustness to adversarial attacks. With up to 78.80% FLOPs reduction and performance on par with state-of-the-art methods across different benchmarks, our approach reduces energy consumption and carbon emissions by up to 66.99% and 68.75%, respectively. Additionally, it avoids shortcut learning and improves robustness by up to 4 percentage points under various adversarial attacks. Overall, the Consensus criterion demonstrates its effectiveness in creating robust, efficient, and environmentally friendly pruned models.