Dive into Layers: Neural Network Capacity Bounding using Algebraic Geometry
This provides a theoretical foundation for architecture selection in neural networks, addressing a key challenge for researchers and practitioners, though it is incremental as it builds on existing algebraic geometry tools.
The paper tackles the problem of neural network architecture selection by linking network capacity to the topological complexity of input data using Betti numbers, showing that network size must be sufficient to match this complexity, with experiments on MNIST verifying the analysis.
The empirical results suggest that the learnability of a neural network is directly related to its size. To mathematically prove this, we borrow a tool in topological algebra: Betti numbers to measure the topological geometric complexity of input data and the neural network. By characterizing the expressive capacity of a neural network with its topological complexity, we conduct a thorough analysis and show that the network's expressive capacity is limited by the scale of its layers. Further, we derive the upper bounds of the Betti numbers on each layer within the network. As a result, the problem of architecture selection of a neural network is transformed to determining the scale of the network that can represent the input data complexity. With the presented results, the architecture selection of a fully connected network boils down to choosing a suitable size of the network such that it equips the Betti numbers that are not smaller than the Betti numbers of the input data. We perform the experiments on a real-world dataset MNIST and the results verify our analysis and conclusion. The code is publicly available.