Effective Neural Network $L_0$ Regularization With BinMask
This work addresses the fundamental problem of L0 regularization for improving neural network generalizability and sparsity, but it is incremental as it builds on existing decoupling techniques.
The paper tackles the problem of L0 regularization for neural networks by proposing BinMask, a simple method using deterministic binary masks and straight-through estimation, which achieves competitive performance on feature selection, network sparsification, and model regularization benchmarks without task-specific tuning.
$L_0$ regularization of neural networks is a fundamental problem. In addition to regularizing models for better generalizability, $L_0$ regularization also applies to selecting input features and training sparse neural networks. There is a large body of research on related topics, some with quite complicated methods. In this paper, we show that a straightforward formulation, BinMask, which multiplies weights with deterministic binary masks and uses the identity straight-through estimator for backpropagation, is an effective $L_0$ regularizer. We evaluate BinMask on three tasks: feature selection, network sparsification, and model regularization. Despite its simplicity, BinMask achieves competitive performance on all the benchmarks without task-specific tuning compared to methods designed for each task. Our results suggest that decoupling weights from mask optimization, which has been widely adopted by previous work, is a key component for effective $L_0$ regularization.