Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
This addresses the depth-width tradeoff problem in neural network design for researchers and practitioners, providing theoretical and empirical evidence for depth advantages, though it is incremental as it builds on existing separation results.
The paper demonstrates that deeper neural networks can better approximate simple natural functions like indicators of balls and ellipses, radial functions, and smooth functions compared to shallower networks, even with larger widths, with experimental validation showing improved learning for a unit ball indicator.
We provide several new depth-based separation results for feed-forward neural networks, proving that various types of simple and natural functions can be better approximated using deeper networks than shallower ones, even if the shallower networks are much larger. This includes indicators of balls and ellipses; non-linear functions which are radial with respect to the $L_1$ norm; and smooth non-linear functions. We also show that these gaps can be observed experimentally: Increasing the depth indeed allows better learning than increasing width, when training neural networks to learn an indicator of a unit ball.