DANTE: Deep AlterNations for Training nEural networks
This addresses the challenge of training neural networks with both differentiable and non-differentiable activation functions, offering an alternative to gradient-based methods, though it appears incremental as it builds on existing optimization principles.
The authors tackled the problem of training neural networks by proposing DANTE, a method based on alternating minimization that casts training as a bi-quasi-convex optimization problem, and found it to be competitive with traditional backpropagation in solution quality and training speed on standard datasets.
We present DANTE, a novel method for training neural networks using the alternating minimization principle. DANTE provides an alternate perspective to traditional gradient-based backpropagation techniques commonly used to train deep networks. It utilizes an adaptation of quasi-convexity to cast training a neural network as a bi-quasi-convex optimization problem. We show that for neural network configurations with both differentiable (e.g. sigmoid) and non-differentiable (e.g. ReLU) activation functions, we can perform the alternations effectively in this formulation. DANTE can also be extended to networks with multiple hidden layers. In experiments on standard datasets, neural networks trained using the proposed method were found to be promising and competitive to traditional backpropagation techniques, both in terms of quality of the solution, as well as training speed.