Learning Two-layer Neural Networks with Symmetric Inputs
This addresses the challenge of parameter recovery in neural networks for researchers, offering a theoretical guarantee under symmetric inputs, though it is incremental as it builds on tensor decomposition methods.
The paper tackles the problem of learning a two-layer neural network from symmetric input distributions, presenting an algorithm that guarantees recovery of the ground-truth parameters using a method-of-moments framework and spectral algorithms, with experiments showing robust learning with a small number of samples.
We give a new algorithm for learning a two-layer neural network under a general class of input distributions. Assuming there is a ground-truth two-layer network $$ y = A σ(Wx) + ξ, $$ where $A,W$ are weight matrices, $ξ$ represents noise, and the number of neurons in the hidden layer is no larger than the input or output, our algorithm is guaranteed to recover the parameters $A,W$ of the ground-truth network. The only requirement on the input $x$ is that it is symmetric, which still allows highly complicated and structured input. Our algorithm is based on the method-of-moments framework and extends several results in tensor decompositions. We use spectral algorithms to avoid the complicated non-convex optimization in learning neural networks. Experiments show that our algorithm can robustly learn the ground-truth neural network with a small number of samples for many symmetric input distributions.