Towards a General Theory of Infinite-Width Limits of Neural Classifiers
This work addresses theoretical gaps in neural network training approximations for researchers in machine learning theory, though it is incremental in refining existing limit theories.
The authors tackled the problem of linking two distinct infinite-width limit theories for neural networks (mean-field and NTK), proposing a general framework that reveals a new discrete-time mean-field limit and demonstrates limitations of existing approximations for finite-width networks, particularly with non-small learning rates.
Obtaining theoretical guarantees for neural networks training appears to be a hard problem in a general case. Recent research has been focused on studying this problem in the limit of infinite width and two different theories have been developed: a mean-field (MF) and a constant kernel (NTK) limit theories. We propose a general framework that provides a link between these seemingly distinct theories. Our framework out of the box gives rise to a discrete-time MF limit which was not previously explored in the literature. We prove a convergence theorem for it and show that it provides a more reasonable approximation for finite-width nets compared to the NTK limit if learning rates are not very small. Also, our framework suggests a limit model that coincides neither with the MF limit nor with the NTK one. We show that for networks with more than two hidden layers RMSProp training has a non-trivial discrete-time MF limit but GD training does not have one. Overall, our framework demonstrates that both MF and NTK limits have considerable limitations in approximating finite-sized neural nets, indicating the need for designing more accurate infinite-width approximations for them.