Adapting Resilient Propagation for Deep Learning
This is an incremental improvement for training deep neural networks, potentially benefiting practitioners in computer vision and related fields.
The paper tackled the problem of standard Resilient Propagation (Rprop) struggling with deep neural networks by modifying it with a dropout technique, resulting in improved learning speed and accuracy on the MNIST dataset.
The Resilient Propagation (Rprop) algorithm has been very popular for backpropagation training of multilayer feed-forward neural networks in various applications. The standard Rprop however encounters difficulties in the context of deep neural networks as typically happens with gradient-based learning algorithms. In this paper, we propose a modification of the Rprop that combines standard Rprop steps with a special drop out technique. We apply the method for training Deep Neural Networks as standalone components and in ensemble formulations. Results on the MNIST dataset show that the proposed modification alleviates standard Rprop's problems demonstrating improved learning speed and accuracy.