Optimal Approximation with Sparse Neural Networks and Applications
This work provides a theoretical framework for understanding neural network approximation, which is incremental as it builds on existing mathematical theories of representation systems.
The paper tackles the problem of measuring function class complexity using sparse neural networks and representation systems, proving a fundamental bound theorem that links intrinsic quantities to approximation ability, and applies this to approximate B-spline functions and analyze β cartoon-like functions with rate-distortion theory.
We use deep sparsely connected neural networks to measure the complexity of a function class in $L^2(\mathbb R^d)$ by restricting connectivity and memory requirement for storing the neural networks. We also introduce representation system - a countable collection of functions to guide neural networks, since approximation theory with representation system has been well developed in Mathematics. We then prove the fundamental bound theorem, implying a quantity intrinsic to the function class itself can give information about the approximation ability of neural networks and representation system. We also provides a method for transferring existing theories about approximation by representation systems to that of neural networks, greatly amplifying the practical values of neural networks. Finally, we use neural networks to approximate B-spline functions, which are used to generate the B-spline curves. Then, we analyse the complexity of a class called $β$ cartoon-like functions using rate-distortion theory and wedgelets construction.