Expressivity of Deep Neural Networks
It synthesizes existing knowledge on neural network expressivity, which is incremental as it reviews prior work without introducing new methods.
This review paper provides a comprehensive overview of approximation results for neural networks, discussing rates for classical function spaces and the benefits of deep over shallow networks for structured function classes, covering feedforward, convolutional, residual, and recurrent architectures.
In this review paper, we give a comprehensive overview of the large variety of approximation results for neural networks. Approximation rates for classical function spaces as well as benefits of deep neural networks over shallow ones for specifically structured function classes are discussed. While the mainbody of existing results is for general feedforward architectures, we also depict approximation results for convolutional, residual and recurrent neural networks.