Normalized Convolutional Neural Network
This addresses normalization challenges in convolutional networks for deep learning practitioners, offering a novel method that is incremental as it adapts existing principles to convolution.
The paper tackles the problem of normalization in convolutional neural networks by introducing a Normalized Convolutional Neural Layer that normalizes rows of the im2col matrix, resulting in consistent outperformance of other batch-independent normalization methods in micro-batch training scenarios.
We introduce a Normalized Convolutional Neural Layer, a novel approach to normalization in convolutional networks. Unlike conventional methods, this layer normalizes the rows of the im2col matrix during convolution, making it inherently adaptive to sliced inputs and better aligned with kernel structures. This distinctive approach differentiates it from standard normalization techniques and prevents direct integration into existing deep learning frameworks optimized for traditional convolution operations. Our method has a universal property, making it applicable to any deep learning task involving convolutional layers. By inherently normalizing within the convolution process, it serves as a convolutional adaptation of Self-Normalizing Networks, maintaining their core principles without requiring additional normalization layers. Notably, in micro-batch training scenarios, it consistently outperforms other batch-independent normalization methods. This performance boost arises from standardizing the rows of the im2col matrix, which theoretically leads to a smoother loss gradient and improved training stability.