BINGO: A Novel Pruning Mechanism to Reduce the Size of Neural Networks
This addresses the high costs and environmental impact of training large models, benefiting companies and researchers, though it is an incremental improvement over existing pruning methods.
The paper tackles the problem of large, expensive neural networks by introducing BINGO, a pruning mechanism that assigns significance scores to weights during training, enabling one-shot pruning and reducing computational costs while preserving accuracy.
Over the past decade, the use of machine learning has increased exponentially. Models are far more complex than ever before, growing to gargantuan sizes and housing millions of weights. Unfortunately, the fact that large models have become the state of the art means that it often costs millions of dollars to train and operate them. These expenses not only hurt companies but also bar non-wealthy individuals from contributing to new developments and force consumers to pay greater prices for AI. Current methods used to prune models, such as iterative magnitude pruning, have shown great accuracy but require an iterative training sequence that is incredibly computationally and environmentally taxing. To solve this problem, BINGO is introduced. BINGO, during the training pass, studies specific subsets of a neural network one at a time to gauge how significant of a role each weight plays in contributing to a network's accuracy. By the time training is done, BINGO generates a significance score for each weight, allowing for insignificant weights to be pruned in one shot. BINGO provides an accuracy-preserving pruning technique that is less computationally intensive than current methods, allowing for a world where AI growth does not have to mean model growth, as well.