Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
This addresses the training inefficiency in large-scale neural networks for machine learning practitioners, offering a novel approach to enhance learning dynamics.
The paper tackles the problem of slow training in infinitely wide neural networks by introducing a finite-sized bottleneck, which enables data-dependent feature learning and accelerates training compared to purely infinite networks, with empirical improvements in overall performance.
We analyze the learning dynamics of infinitely wide neural networks with a finite sized bottle-neck. Unlike the neural tangent kernel limit, a bottleneck in an otherwise infinite width network al-lows data dependent feature learning in its bottle-neck representation. We empirically show that a single bottleneck in infinite networks dramatically accelerates training when compared to purely in-finite networks, with an improved overall performance. We discuss the acceleration phenomena by drawing similarities to infinitely wide deep linear models, where the acceleration effect of a bottleneck can be understood theoretically.