A Theoretical Study of Neural Network Expressive Power via Manifold Topology
This work addresses a theoretical challenge in machine learning by providing insights into network design based on manifold properties, though it appears incremental as it builds on existing geometric studies.
The authors tackled the problem of determining the required size of neural networks for data on low-dimensional manifolds by integrating topological and geometric aspects, resulting in a derived upper bound for ReLU neural network size.
A prevalent assumption regarding real-world data is that it lies on or close to a low-dimensional manifold. When deploying a neural network on data manifolds, the required size, i.e., the number of neurons of the network, heavily depends on the intricacy of the underlying latent manifold. While significant advancements have been made in understanding the geometric attributes of manifolds, it's essential to recognize that topology, too, is a fundamental characteristic of manifolds. In this study, we investigate network expressive power in terms of the latent data manifold. Integrating both topological and geometric facets of the data manifold, we present a size upper bound of ReLU neural networks.