Expected Gradients of Maxout Networks and Consequences to Parameter Initialization
This provides a practical solution for training deep maxout networks, though it appears incremental relative to existing initialization methods for other architectures.
The authors tackled the problem of unstable parameter initialization in maxout networks by analyzing gradient moments and developing initialization strategies that prevent vanishing/exploding gradients. Their approach improved SGD and Adam training of deep maxout networks in experiments.
We study the gradients of a maxout network with respect to inputs and parameters and obtain bounds for the moments depending on the architecture and the parameter distribution. We observe that the distribution of the input-output Jacobian depends on the input, which complicates a stable parameter initialization. Based on the moments of the gradients, we formulate parameter initialization strategies that avoid vanishing and exploding gradients in wide networks. Experiments with deep fully-connected and convolutional networks show that this strategy improves SGD and Adam training of deep maxout networks. In addition, we obtain refined bounds on the expected number of linear regions, results on the expected curve length distortion, and results on the NTK.