Function Approximation in Numerically Rank-Deficient Bases
This addresses numerical stability issues in computational mathematics and machine learning for researchers and practitioners using function approximation methods.
The paper tackles the problem of linear function approximation in finite-precision arithmetic with non-orthogonal bases, showing that rounding errors distort the spanned space and that regularization controls error amplification and relaxes sampling requirements for accurate least squares approximation.
We study linear function approximation in a finite basis under finite-precision arithmetic. In a highly non-orthogonal basis, certain directions are only weakly represented, so that rounding errors can significantly distort the effectively spanned space. In the first part of the paper, we formalize this phenomenon through the notion of a numerical span. Using a novel model for the rounding errors involved, we prove that approximation in the numerical span behaves like approximation in exact arithmetic subject to an additional penalty proportional to the size of the expansion coefficients and the unit roundoff. A key implication is that straightforward numerical orthogonalization cannot mitigate the effects induced by finite-precision arithmetic. The framework also provides a theoretical justification for $\ell^2$-regularized approximation. Moreover, regularization controls the amplification of rounding errors in the computation of expansion coefficients. In the second part of the paper, we address sampling for function approximation in the presence of numerical rank-deficiency. We demonstrate that regularization has another fundamental benefit: it relaxes the conditions required for accurate least squares approximation from sampled data. This effect is made concrete through an analysis of randomized sampling based on a regularized variant of the Christoffel function. The resulting sample complexity bounds depend on an effective dimension that measures the number of directions that remain useful after finite-precision rounding. We also show that regularization renders the Christoffel function computable in contrast to the standard Christoffel function, whose numerical evaluation may require arbitrarily high precision in the presence of numerical rank-deficiency. We apply the derived theory to obtain new results for the discretization of univariate Fourier extension frames.