Approximation of Functions: Optimal Sampling and Complexity
This is an incremental survey article summarizing existing research on function approximation for applications in machine learning and numerical analysis.
This paper examines the fundamental limits and optimal strategies for approximating functions from finite evaluations, addressing both information-theoretic constraints and algorithmic approaches to achieve near-optimal performance.
We consider approximation or recovery of functions based on a finite number of function evaluations. This is a well-studied problem in optimal recovery, machine learning, and numerical analysis in general, but many fundamental insights were obtained only recently. We discuss different aspects of the information-theoretic limit that appears because of the limited amount of data available, as well as algorithms and sampling strategies that come as close to it as possible. We also discuss (optimal) sampling in a broader sense, allowing other types of measurements that may be nonlinear, adaptive and random, and present several relations between the different settings in the spirit of information-based complexity. We hope that this article provides both, a basic introduction to the subject and a contemporary summary of the current state of research.