The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information
This work addresses the need for more efficient and theoretically grounded sparse neural networks, which is crucial for reducing computational and memory costs in machine learning applications.
The paper tackles the problem of improving sparse recovery algorithms for deep neural networks by connecting classical Optimal Brain Surgeon (OBS) pruning with iterative sparse recovery methods, resulting in improved convergence bounds and strong practical performance on Transformer models for vision and language tasks.
The rising footprint of machine learning has led to a focus on imposing \emph{model sparsity} as a means of reducing computational and memory costs. For deep neural networks (DNNs), the state-of-the-art accuracy-vs-sparsity is achieved by heuristics inspired by the classical Optimal Brain Surgeon (OBS) framework~\citep{lecun90brain, hassibi1992second, hassibi1993optimal}, which leverages loss curvature information to make better pruning decisions. Yet, these results still lack a solid theoretical understanding, and it is unclear whether they can be improved by leveraging connections to the wealth of work on sparse recovery algorithms. In this paper, we draw new connections between these two areas and present new sparse recovery algorithms inspired by the OBS framework that comes with theoretical guarantees under reasonable assumptions and have strong practical performance. Specifically, our work starts from the observation that we can leverage curvature information in OBS-like fashion upon the projection step of classic iterative sparse recovery algorithms such as IHT. We show for the first time that this leads both to improved convergence bounds under standard assumptions. Furthermore, we present extensions of this approach to the practical task of obtaining accurate sparse DNNs, and validate it experimentally at scale for Transformer-based models on vision and language tasks.