Neural Tangent Kernel Analysis of Deep Narrow Neural Networks
This work addresses a foundational gap in theoretical machine learning by extending analysis to deep narrow networks, which is incremental as it builds on existing NTK frameworks.
The paper tackles the lack of theoretical understanding of depth's role in deep learning by providing the first trainability guarantee for infinitely deep but narrow neural networks, establishing this using Neural Tangent Kernel theory for both multilayer perceptrons and convolutional neural networks.
The tremendous recent progress in analyzing the training dynamics of overparameterized neural networks has primarily focused on wide networks and therefore does not sufficiently address the role of depth in deep learning. In this work, we present the first trainability guarantee of infinitely deep but narrow neural networks. We study the infinite-depth limit of a multilayer perceptron (MLP) with a specific initialization and establish a trainability guarantee using the NTK theory. We then extend the analysis to an infinitely deep convolutional neural network (CNN) and perform brief experiments.