Sharp analysis of sketched least squares and randomized low-rank approximation
Provides theoretical foundations for choosing random embeddings in two fundamental randomized algorithms, offering guidance for practitioners and establishing optimality guarantees.
This paper identifies the optimal random embeddings for sketch-and-solve least squares and randomized SVD, proving that random orthonormal matrices are minimax optimal for the former and rotation-invariant embeddings for the latter, and derives sharp error bounds.
Two widely used randomized algorithms are the sketch-and-solve method for least-squares regression and the randomized SVD for low-rank approximation. These algorithms apply a random embedding to compress a target matrix, and they perform computations on the compressed matrix to save computational cost. This paper asks, what is the optimal random embedding in these algorithms? Also, what is the sharpest possible error bound for the optimal embedding? The paper proves that a random orthonormal matrix is minimax optimal for the sketch-and-solve algorithm while any rotation-invariant embedding is minimax optimal for the randomized SVD. Following these results, the paper obtains the best possible error bounds for sketched least-squares and the randomized SVD. Last, empirical experiments provide evidence of universality phenomena, in which several random embeddings lead to similar accuracy to the optimal embeddings in practice.