MMA Regularization: Decorrelating Weights of Neural Networks by Maximizing the Minimal Angles
This addresses generalization issues in neural networks for machine learning practitioners, but it is incremental as it builds on existing regularization methods.
The paper tackles the problem of strong correlation between neurons or filters weakening neural network generalization by proposing MMA regularization, which maximizes minimal pairwise angles to decorrelate weights, achieving performance improvements on datasets like CIFAR100 and TinyImageNet.
The strong correlation between neurons or filters can significantly weaken the generalization ability of neural networks. Inspired by the well-known Tammes problem, we propose a novel diversity regularization method to address this issue, which makes the normalized weight vectors of neurons or filters distributed on a hypersphere as uniformly as possible, through maximizing the minimal pairwise angles (MMA). This method can easily exert its effect by plugging the MMA regularization term into the loss function with negligible computational overhead. The MMA regularization is simple, efficient, and effective. Therefore, it can be used as a basic regularization method in neural network training. Extensive experiments demonstrate that MMA regularization is able to enhance the generalization ability of various modern models and achieves considerable performance improvements on CIFAR100 and TinyImageNet datasets. In addition, experiments on face verification show that MMA regularization is also effective for feature learning. Code is available at: https://github.com/wznpub/MMA_Regularization.