Convex Relaxations of Convolutional Neural Nets
This work addresses the optimization challenges in neural networks for researchers, offering a theoretical guarantee for global convergence in specific cases, though it is incremental as it applies to a limited network architecture.
The authors tackled the problem of non-convex optimization in convolutional neural networks by proposing convex relaxations for one-hidden-layer networks with fixed output weights, proving that under a planted model with Gaussian data, the relaxation recovers the global minimum given enough samples and identifying a phase transition in this recovery.
We propose convex relaxations for convolutional neural nets with one hidden layer where the output weights are fixed. For convex activation functions such as rectified linear units, the relaxations are convex second order cone programs which can be solved very efficiently. We prove that the relaxation recovers the global minimum under a planted model assumption, given sufficiently many training samples from a Gaussian distribution. We also identify a phase transition phenomenon in recovering the global minimum for the relaxation.