What makes a good feedforward computational graph?
This work addresses the impact of graph structure on neural network performance, focusing on feedforward graphs, but it is incremental as it builds on existing studies of graph properties.
The paper investigates desirable properties of feedforward computational graphs, identifying fidelity and mixing time as key measures, and evaluates popular graphs through theoretical analysis and correlation with neural network performance.
As implied by the plethora of literature on graph rewiring, the choice of computational graph employed by a neural network can make a significant impact on its downstream performance. Certain effects related to the computational graph, such as under-reaching and over-squashing, may even render the model incapable of learning certain functions. Most of these effects have only been thoroughly studied in the domain of undirected graphs; however, recent years have seen a significant rise in interest in feedforward computational graphs: directed graphs without any back edges. In this paper, we study the desirable properties of a feedforward computational graph, discovering two important complementary measures: fidelity and mixing time, and evaluating a few popular choices of graphs through the lens of these measures. Our study is backed by both theoretical analyses of the metrics' asymptotic behaviour for various graphs, as well as correlating these metrics to the performance of trained neural network models using the corresponding graphs.