Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture
This provides a foundational mathematical insight into neural network theory, affecting all of ML/AI, but it is incremental as it builds on prior work.
The paper tackles the problem of understanding the behavior of neural networks with arbitrary directed acyclic graph architectures as their width increases, showing that they undergo a transition to linearity. The result generalizes recent works on transition to linearity or constancy of the Neural Tangent Kernel for standard architectures.
In this paper we show that feedforward neural networks corresponding to arbitrary directed acyclic graphs undergo transition to linearity as their "width" approaches infinity. The width of these general networks is characterized by the minimum in-degree of their neurons, except for the input and first layers. Our results identify the mathematical structure underlying transition to linearity and generalize a number of recent works aimed at characterizing transition to linearity or constancy of the Neural Tangent Kernel for standard architectures.