On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity
This work addresses the need for better interpretability and effectiveness in regularization for neural networks, particularly in noisy or data-scarce environments, though it appears incremental as it builds on existing regularization principles.
The authors tackled the problem of overfitting in neural networks by introducing Model Gradient Similarity (MGS), a framework that serves as a metric for regularization, explains how different regularizers work via a common mechanism, and enables a new regularization scheme that shows excellent performance, especially in challenging settings like high label noise or limited data.
Most complex machine learning and modelling techniques are prone to over-fitting and may subsequently generalise poorly to future data. Artificial neural networks are no different in this regard and, despite having a level of implicit regularisation when trained with gradient descent, often require the aid of explicit regularisers. We introduce a new framework, Model Gradient Similarity (MGS), that (1) serves as a metric of regularisation, which can be used to monitor neural network training, (2) adds insight into how explicit regularisers, while derived from widely different principles, operate via the same mechanism underneath by increasing MGS, and (3) provides the basis for a new regularisation scheme which exhibits excellent performance, especially in challenging settings such as high levels of label noise or limited sample sizes.