Analyzing Neural Networks Based on Random Graphs
This work addresses the problem of understanding and optimizing neural network architectures for researchers in machine learning, though it is incremental as it builds on existing graph-based analysis methods.
The study evaluated neural networks with random graph architectures and found that classical graph invariants alone cannot identify top-performing networks, leading to the introduction of a new characteristic that selects quasi-1-dimensional graphs, which constitute a majority of the best performers, with networks having short-range connections and many resolution-reducing pathways showing better accuracy.
We perform a massive evaluation of neural networks with architectures corresponding to random graphs of various types. We investigate various structural and numerical properties of the graphs in relation to neural network test accuracy. We find that none of the classical numerical graph invariants by itself allows to single out the best networks. Consequently, we introduce a new numerical graph characteristic that selects a set of quasi-1-dimensional graphs, which are a majority among the best performing networks. We also find that networks with primarily short-range connections perform better than networks which allow for many long-range connections. Moreover, many resolution reducing pathways are beneficial. We provide a dataset of 1020 graphs and the test accuracies of their corresponding neural networks at https://github.com/rmldj/random-graph-nn-paper