Regularization for convolutional kernel tensors to avoid unstable gradient problem in convolutional neural networks
This work aims to improve the stability of training for convolutional neural networks by mitigating exploding/vanishing gradients, which is a common problem for practitioners and researchers in deep learning.
This paper addresses the unstable gradient problem in convolutional neural networks by proposing three new regularization terms for convolutional kernel tensors. These terms constrain the singular values of the transformation matrices associated with each convolution, aiming to prevent them from becoming excessively large or small during training.
Convolutional neural networks are very popular nowadays. Training neural networks is not an easy task. Each convolution corresponds to a structured transformation matrix. In order to help avoid the exploding/vanishing gradient problem, it is desirable that the singular values of each transformation matrix are not large/small in the training process. We propose three new regularization terms for a convolutional kernel tensor to constrain the singular values of each transformation matrix. We show how to carry out the gradient type methods, which provides new insight about the training of convolutional neural networks.