Training of Deep Neural Networks based on Distance Measures using RMSProp
This addresses a fundamental bottleneck in deep learning for researchers and practitioners, though it appears incremental as it builds on existing network structures with a new training approach.
The paper tackles the vanishing gradient problem in deep neural networks by proposing networks based on distance measures and Gaussian activations, showing that using RMSProp enables efficient training and reduces gradient issues even in deep networks.
The vanishing gradient problem was a major obstacle for the success of deep learning. In recent years it was gradually alleviated through multiple different techniques. However the problem was not really overcome in a fundamental way, since it is inherent to neural networks with activation functions based on dot products. In a series of papers, we are going to analyze alternative neural network structures which are not based on dot products. In this first paper, we revisit neural networks built up of layers based on distance measures and Gaussian activation functions. These kinds of networks were only sparsely used in the past since they are hard to train when using plain stochastic gradient descent methods. We show that by using Root Mean Square Propagation (RMSProp) it is possible to efficiently learn multi-layer neural networks. Furthermore we show that when appropriately initialized these kinds of neural networks suffer much less from the vanishing and exploding gradient problem than traditional neural networks even for deep networks.