Convexifying Sparse Interpolation with Infinitely Wide Neural Networks: An Atomic Norm Approach
This work addresses interpolation problems in neural networks for researchers in optimization and machine learning, but it is incremental as it builds on existing atomic norm frameworks.
The paper tackles exact data interpolation with sparse, infinitely wide neural networks using leaky ReLU activations by deriving convex hull characterizations via atomic norms, leading to equivalent convex formulations, and shows experimental efficacy compared to gradient descent training.
This work examines the problem of exact data interpolation via sparse (neuron count), infinitely wide, single hidden layer neural networks with leaky rectified linear unit activations. Using the atomic norm framework of [Chandrasekaran et al., 2012], we derive simple characterizations of the convex hulls of the corresponding atomic sets for this problem under several different constraints on the weights and biases of the network, thus obtaining equivalent convex formulations for these problems. A modest extension of our proposed framework to a binary classification problem is also presented. We explore the efficacy of the resulting formulations experimentally, and compare with networks trained via gradient descent.