Learning One-hidden-layer ReLU Networks via Gradient Descent
This provides theoretical guarantees for practical training of simple neural networks, addressing a foundational issue in machine learning, though it is incremental as it builds on prior work for specific network architectures.
The paper tackles the problem of learning one-hidden-layer ReLU neural networks from noisy data, proving that gradient descent with tensor initialization converges linearly to the ground-truth parameters up to statistical error, with numerical experiments supporting the theory.
We study the problem of learning one-hidden-layer neural networks with Rectified Linear Unit (ReLU) activation function, where the inputs are sampled from standard Gaussian distribution and the outputs are generated from a noisy teacher network. We analyze the performance of gradient descent for training such kind of neural networks based on empirical risk minimization, and provide algorithm-dependent guarantees. In particular, we prove that tensor initialization followed by gradient descent can converge to the ground-truth parameters at a linear rate up to some statistical error. To the best of our knowledge, this is the first work characterizing the recovery guarantee for practical learning of one-hidden-layer ReLU networks with multiple neurons. Numerical experiments verify our theoretical findings.