Sparse-Aware Neural Networks for Nonlinear Functionals: Mitigating the Exponential Dependence on Dimension
This work addresses the curse of dimensionality in functional learning, a key issue in operator learning for fields like scientific computing, but it is incremental as it builds on existing theories by incorporating sparsity.
The paper tackles the problem of learning nonlinear functionals from discrete samples, which suffers from exponential dependence on dimension and limited interpretability, by proposing a sparse-aware neural network framework that improves approximation rates and reduces sample sizes in function spaces like those with fast frequency decay and mixed smoothness.
Deep neural networks have emerged as powerful tools for learning operators defined over infinite-dimensional function spaces. However, existing theories frequently encounter difficulties related to dimensionality and limited interpretability. This work investigates how sparsity can help address these challenges in functional learning, a central ingredient in operator learning. We propose a framework that employs convolutional architectures to extract sparse features from a finite number of samples, together with deep fully connected networks to effectively approximate nonlinear functionals. Using universal discretization methods, we show that sparse approximators enable stable recovery from discrete samples. In addition, both the deterministic and the random sampling schemes are sufficient for our analysis. These findings lead to improved approximation rates and reduced sample sizes in various function spaces, including those with fast frequency decay and mixed smoothness. They also provide new theoretical insights into how sparsity can alleviate the curse of dimensionality in functional learning.