Benefits of depth in neural networks
This work provides a theoretical foundation for the importance of depth in neural networks, addressing a fundamental question in machine learning for researchers and practitioners.
The paper tackles the problem of understanding the benefits of depth in neural networks by proving that deep networks with a cubic number of layers cannot be approximated by shallower networks without an exponential increase in size, specifically requiring at least 2^k nodes for networks with O(k) layers.
For any positive integer $k$, there exist neural networks with $Θ(k^3)$ layers, $Θ(1)$ nodes per layer, and $Θ(1)$ distinct parameters which can not be approximated by networks with $\mathcal{O}(k)$ layers unless they are exponentially large --- they must possess $Ω(2^k)$ nodes. This result is proved here for a class of nodes termed "semi-algebraic gates" which includes the common choices of ReLU, maximum, indicator, and piecewise polynomial functions, therefore establishing benefits of depth against not just standard networks with ReLU gates, but also convolutional networks with ReLU and maximization gates, sum-product networks, and boosted decision trees (in this last case with a stronger separation: $Ω(2^{k^3})$ total tree nodes are required).