Neural networks: deep, shallow, or in between?
This work addresses theoretical limits in neural network design for researchers in approximation theory and machine learning, providing foundational insights into depth vs. width trade-offs.
The paper investigates the approximation error of feed-forward neural networks with varying width and depth, showing that significant improvements over entropy number rates require infinite depth, while increasing width alone yields no benefit.
We give estimates from below for the error of approximation of a compact subset from a Banach space by the outputs of feed-forward neural networks with width W, depth l and Lipschitz activation functions. We show that, modulo logarithmic factors, rates better that entropy numbers' rates are possibly attainable only for neural networks for which the depth l goes to infinity, and that there is no gain if we fix the depth and let the width W go to infinity.