Learning Sparse Networks Using Targeted Dropout
This addresses the need for more efficient neural network models in machine learning by enabling easier pruning without complex regularization, though it is incremental as it builds on existing dropout and pruning techniques.
The paper tackles the problem of training neural networks that are robust to pruning by introducing targeted dropout, a method that stochastically drops units or weights during training based on a sparsity criterion, resulting in networks that can be effectively pruned post-training.
Neural networks are easier to optimise when they have many more weights than are required for modelling the mapping from inputs to outputs. This suggests a two-stage learning procedure that first learns a large net and then prunes away connections or hidden units. But standard training does not necessarily encourage nets to be amenable to pruning. We introduce targeted dropout, a method for training a neural network so that it is robust to subsequent pruning. Before computing the gradients for each weight update, targeted dropout stochastically selects a set of units or weights to be dropped using a simple self-reinforcing sparsity criterion and then computes the gradients for the remaining weights. The resulting network is robust to post hoc pruning of weights or units that frequently occur in the dropped sets. The method improves upon more complicated sparsifying regularisers while being simple to implement and easy to tune.