WLD-Reg: A Data-dependent Within-layer Diversity Regularizer
This addresses the issue of neuron redundancy in neural networks for machine learning practitioners, though it is incremental as it builds on existing regularization techniques.
The paper tackles the problem of limited feedback in neural network training by introducing a within-layer diversity regularizer that encourages activation diversity within layers, resulting in enhanced performance across multiple state-of-the-art models in various tasks.
Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization, where the errors are back-propagated from the last layer back to the first one. At each optimization step, neurons at a given layer receive feedback from neurons belonging to higher layers of the hierarchy. In this paper, we propose to complement this traditional 'between-layer' feedback with additional 'within-layer' feedback to encourage the diversity of the activations within the same layer. To this end, we measure the pairwise similarity between the outputs of the neurons and use it to model the layer's overall diversity. We present an extensive empirical study confirming that the proposed approach enhances the performance of several state-of-the-art neural network models in multiple tasks. The code is publically available at \url{https://github.com/firasl/AAAI-23-WLD-Reg}