Solution space and storage capacity of fully connected two-layer neural networks with generic activation functions
This work provides theoretical insights into neural network expressiveness, potentially aiding architecture selection, but it is incremental as it builds on existing statistical physics methods.
The study analyzed the solution space and storage capacity of fully connected two-layer neural networks with generic activation functions, finding that storage capacity per parameter remains finite even at infinite width and that increasing dataset size triggers a phase transition where solution space splits into disjoint regions.
The storage capacity of a binary classification model is the maximum number of random input-output pairs per parameter that the model can learn. It is one of the indicators of the expressive power of machine learning models and is important for comparing the performance of various models. In this study, we analyze the structure of the solution space and the storage capacity of fully connected two-layer neural networks with general activation functions using the replica method from statistical physics. Our results demonstrate that the storage capacity per parameter remains finite even with infinite width and that the weights of the network exhibit negative correlations, leading to a 'division of labor'. In addition, we find that increasing the dataset size triggers a phase transition at a certain transition point where the permutation symmetry of weights is broken, resulting in the solution space splitting into disjoint regions. We identify the dependence of this transition point and the storage capacity on the choice of activation function. These findings contribute to understanding the influence of activation functions and the number of parameters on the structure of the solution space, potentially offering insights for selecting appropriate architectures based on specific objectives.