Defining Neural Network Architecture through Polytope Structures of Dataset
This provides theoretical insights into neural network design for researchers, though it is incremental as it builds on existing geometric approaches.
The paper tackles the unclear relationship between dataset complexity and neural network architecture by defining width bounds based on dataset polytope structures, showing that datasets like MNIST, Fashion-MNIST, and CIFAR10 can be efficiently represented with no more than two polytopes and a small number of faces.
Current theoretical and empirical research in neural networks suggests that complex datasets require large network architectures for thorough classification, yet the precise nature of this relationship remains unclear. This paper tackles this issue by defining upper and lower bounds for neural network widths, which are informed by the polytope structure of the dataset in question. We also delve into the application of these principles to simplicial complexes and specific manifold shapes, explaining how the requirement for network width varies in accordance with the geometric complexity of the dataset. Moreover, we develop an algorithm to investigate a converse situation where the polytope structure of a dataset can be inferred from its corresponding trained neural networks. Through our algorithm, it is established that popular datasets such as MNIST, Fashion-MNIST, and CIFAR10 can be efficiently encapsulated using no more than two polytopes with a small number of faces.