Approximation smooth and sparse functions by deep neural networks without saturation
This work addresses a foundational problem in approximation theory for machine learning, providing theoretical insights into the advantages of depth in neural networks.
The authors tackled the problem of approximating smooth and sparse functions using deep neural networks, proving that a three-hidden-layer network achieves optimal approximation rates with controllable parameters and that adding one more hidden layer avoids saturation issues.
Constructing neural networks for function approximation is a classical and longstanding topic in approximation theory. In this paper, we aim at constructing deep neural networks (deep nets for short) with three hidden layers to approximate smooth and sparse functions. In particular, we prove that the constructed deep nets can reach the optimal approximation rate in approximating both smooth and sparse functions with controllable magnitude of free parameters. Since the saturation that describes the bottleneck of approximate is an insurmountable problem of constructive neural networks, we also prove that deepening the neural network with only one more hidden layer can avoid the saturation. The obtained results underlie advantages of deep nets and provide theoretical explanations for deep learning.