John Svadlenka

Victor Y. Pan, Qi Luan, John Svadlenka et al.

We study superfast algorithms that computes low rank approximation of a matrix (hereafter referred to as LRA) that use much fewer memory cells and arithmetic operations than the input matrix has entries. We first specify a family of 2mn matrices of size m*n such that for almost 50% of them any superfast LRA algorithm fails to improve the poor trivial approximation by the matrix filled with zeros, but then we prove that the class of all such hard inputs is narrow - the cross-approximation (hereafter {C-A}) superfast iterations as well as some more primitive superfast algorithms compute reasonably accurate LRAs in their transparent CUR form (i) to any matrix allowing close LRA except for small norm perturbations of matrices of an algebraic variety of a smaller dimension, (ii) to the average matrix allowing close LRA, (iii) to the average sparse matrix allowing close LRA and (iv) with a high probability to any matrix allowing close LRA if it is pre-processed fast with a random Gaussian, SRHT or SRFT multiplier. Moreover empirically the output LRAs remain accurate when we perform the computations superfast by replacing such a multiplier with one of our sparse and structured multipliers. Our techniques, auxiliary results and extensions may be of some independent interest. We analyze C-A and other superfast algorithms twice -- based on two well-known sufficient criteria for obtaining accurate LRAs. We provide a distinct proof in the case of superfast variant of randomized algorithms of [DMM08], improve a decade-old estimate for the norm of the inverse of a Gaussian matrix, prove such an estimate also in the case of a sparse Gaussian matrix, present some novel advanced pre-processing techniques for fast and superfast computation of LRA, and extend our results to dramatic acceleration of the Fast Multipole Method (FMM) and the Conjugate Gradient algorithms.

Victor Y. Pan, Qi Luan, John Svadlenka et al.

Low Rank Approximation is among most fundamental subjects of numerical linear algebra having important applications to various areas of modern computing and %they range from machine learning theory and %neural networks to data mining and analysis. The known algorithms compute such approximations by using more flops than the input matrix has entries, but we prove that much fewer flops than entries are sufficient in the case of the average input ("flop" stands for "floating point arithmetic operation"). We prove this twice -- for the solutions by means of two distinct algorithms, and we analyze them by applying two different approaches. Our analysis of both algorithms is quite involved, but we devise them mostly by simplifying, combining, and ameliorating the known techniques, although we propose some technical novelties for further enhancing the performance of the popular Cross-Approximation Algorithms. They are highly efficient empirically, and we prove that they are efficient for the average input. We specify some narrow classes of hard inputs for which the presented algorithms fail with high probability even when we randomize them, but we narrow such classes further by means of preprocessing with new sparse and structured multipliers. The average complexity estimates do not cover many realistic input classes, but our formal analysis is in good accordance with the results of our tests applied to benchmark inputs from discretized PDEs and Integral quations and to random inputs. Our work should already be of practical value but also leads to research challenges. At the end we list some of them, propose two novel extensions of our progress -- to the acceleration of the Fast Multipole Method and Conjugate Gradient algorithms, and explore and slightly extend the recent techniques of Osinsky, which enhance the output accuracy of CUR Approximation.

Victor Pan, John Svadlenka, Liang Zhao

Low-rank approximation of a matrix by means of structured random sampling has been consistently efficient in its extensive empirical studies around the globe, but adequate formal support for this empirical phenomenon has been missing so far. Based on our novel insight into the subject, we provide such an elusive formal support and derandomize and simplify the known numerical algorithms for low-rank approximation and related computations. Our techniques can be applied to some other areas of fundamental matrix computations, in particular to the Least Squares Regression, Gaussian elimination with no pivoting and block Gaussian elimination. Our formal results and our numerical tests are in good accordance with each other.