Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning
This work addresses the need for more effective pruning methods in deep learning, particularly for achieving high sparsity without performance loss, though it is incremental as it builds on existing magnitude-based pruning.
The paper tackles the problem of pruning neural networks by proposing lookahead pruning, which extends single-layer optimization to multi-layer optimization, and it consistently outperforms magnitude-based pruning, especially at high sparsity levels, on networks like VGG and ResNet.
Magnitude-based pruning is one of the simplest methods for pruning neural networks. Despite its simplicity, magnitude-based pruning and its variants demonstrated remarkable performances for pruning modern architectures. Based on the observation that magnitude-based pruning indeed minimizes the Frobenius distortion of a linear operator corresponding to a single layer, we develop a simple pruning method, coined lookahead pruning, by extending the single layer optimization to a multi-layer optimization. Our experimental results demonstrate that the proposed method consistently outperforms magnitude-based pruning on various networks, including VGG and ResNet, particularly in the high-sparsity regime. See https://github.com/alinlab/lookahead_pruning for codes.