Sharp asymptotics on the compression of two-layer neural networks
This work addresses the compression of neural networks for efficiency in deployment, but it is incremental as it builds on existing mean-field and high-dimensional probability tools.
The paper tackles the problem of compressing an over-parameterized two-layer neural network into a smaller one by minimizing the population L2 loss, showing that in the mean-field limit, the optimal compressed network depends only on expected scaling factors and not on the specific target network realization. For ReLU activations, they conjecture that the optimum is achieved using weights on an Equiangular Tight Frame, supported by numerical evidence.
In this paper, we study the compression of a target two-layer neural network with N nodes into a compressed network with M<N nodes. More precisely, we consider the setting in which the weights of the target network are i.i.d. sub-Gaussian, and we minimize the population L_2 loss between the outputs of the target and of the compressed network, under the assumption of Gaussian inputs. By using tools from high-dimensional probability, we show that this non-convex problem can be simplified when the target network is sufficiently over-parameterized, and provide the error rate of this approximation as a function of the input dimension and N. In this mean-field limit, the simplified objective, as well as the optimal weights of the compressed network, does not depend on the realization of the target network, but only on expected scaling factors. Furthermore, for networks with ReLU activation, we conjecture that the optimum of the simplified optimization problem is achieved by taking weights on the Equiangular Tight Frame (ETF), while the scaling of the weights and the orientation of the ETF depend on the parameters of the target network. Numerical evidence is provided to support this conjecture.