Training Deep Neural Networks via Optimization Over Graphs
This work addresses training efficiency and robustness for deep learning practitioners, but it appears incremental as it adapts existing optimization methods to neural networks.
The paper tackles the problem of training deep neural networks by proposing a distributed optimization approach over graphs, using ADMM to update weights layerwise, and reports that this method is less sensitive to overfitting compared to SGD and Adam.
In this work, we propose to train a deep neural network by distributed optimization over a graph. Two nonlinear functions are considered: the rectified linear unit (ReLU) and a linear unit with both lower and upper cutoffs (DCutLU). The problem reformulation over a graph is realized by explicitly representing ReLU or DCutLU using a set of slack variables. We then apply the alternating direction method of multipliers (ADMM) to update the weights of the network layerwise by solving subproblems of the reformulated problem. Empirical results suggest that the ADMM-based method is less sensitive to overfitting than the stochastic gradient descent (SGD) and Adam methods.