Semi-Implicit Back Propagation
This work addresses training challenges like gradient vanishing and small step sizes for neural network practitioners, but it appears incremental as it builds on existing back propagation and proximal methods.
The authors tackled the slow convergence and instability of stochastic gradient descent in neural network training by proposing a semi-implicit back propagation method, which demonstrated better performance in loss decreasing and accuracy on MNIST and CIFAR-10 datasets compared to SGD and ProxBP.
Neural network has attracted great attention for a long time and many researchers are devoted to improve the effectiveness of neural network training algorithms. Though stochastic gradient descent (SGD) and other explicit gradient-based methods are widely adopted, there are still many challenges such as gradient vanishing and small step sizes, which leads to slow convergence and instability of SGD algorithms. Motivated by error back propagation (BP) and proximal methods, we propose a semi-implicit back propagation method for neural network training. Similar to BP, the difference on the neurons are propagated in a backward fashion and the parameters are updated with proximal mapping. The implicit update for both hidden neurons and parameters allows to choose large step size in the training algorithm. Finally, we also show that any fixed point of convergent sequences produced by this algorithm is a stationary point of the objective loss function. The experiments on both MNIST and CIFAR-10 demonstrate that the proposed semi-implicit BP algorithm leads to better performance in terms of both loss decreasing and training/validation accuracy, compared to SGD and a similar algorithm ProxBP.