On neural network kernels and the storage capacity problem
This work bridges theoretical frameworks for researchers in statistical mechanics and neural network theory, but it is incremental as it reifies an existing connection.
The paper connects the storage capacity problem in wide two-layer neural networks to kernel limits, showing that the effective order parameter from statistical mechanics is equivalent to the infinite-width Neural Network Gaussian Process Kernel, linking expressivity and trainability.
In this short note, we reify the connection between work on the storage capacity problem in wide two-layer treelike neural networks and the rapidly-growing body of literature on kernel limits of wide neural networks. Concretely, we observe that the "effective order parameter" studied in the statistical mechanics literature is exactly equivalent to the infinite-width Neural Network Gaussian Process Kernel. This correspondence connects the expressivity and trainability of wide two-layer neural networks.