Volumization as a Natural Generalization of Weight Decay
This addresses the problem of regularization in neural networks for researchers and practitioners, but it appears incremental as it builds upon existing weight decay methods.
The paper introduces volumization, a novel regularization method for neural networks that generalizes weight decay by interpolating between L2 and L∞ regularization, and demonstrates its effectiveness in improving generalization and preventing memorization in scenarios where standard weight decay works well.
We propose a novel regularization method, called \textit{volumization}, for neural networks. Inspired by physics, we define a physical volume for the weight parameters in neural networks, and we show that this method is an effective way of regularizing neural networks. Intuitively, this method interpolates between an $L_2$ and $L_\infty$ regularization. Therefore, weight decay and weight clipping become special cases of the proposed algorithm. We prove, on a toy example, that the essence of this method is a regularization technique to control bias-variance tradeoff. The method is shown to do well in the categories where the standard weight decay method is shown to work well, including improving the generalization of networks and preventing memorization. Moreover, we show that the volumization might lead to a simple method for training a neural network whose weight is binary or ternary.