Provable wavelet-based neural approximation
This provides theoretical guarantees for neural network approximation, which is foundational for machine learning researchers and practitioners.
The paper tackles the problem of understanding when neural networks can universally approximate functions by developing a wavelet-based theoretical framework that provides sufficient conditions on activation functions for approximation, with explicit error estimates. The result accommodates both smooth and non-smooth activation functions, offering increased flexibility in network design.
In this paper, we develop a wavelet-based theoretical framework for analyzing the universal approximation capabilities of neural networks over a wide range of activation functions. Leveraging wavelet frame theory on the spaces of homogeneous type, we derive sufficient conditions on activation functions to ensure that the associated neural network approximates any functions in the given space, along with an error estimate. These sufficient conditions accommodate a variety of smooth activation functions, including those that exhibit oscillatory behavior. Furthermore, by considering the $L^2$-distance between smooth and non-smooth activation functions, we establish a generalized approximation result that is applicable to non-smooth activations, with the error explicitly controlled by this distance. This provides increased flexibility in the design of network architectures.