Random Quadratic Forms with Dependence: Applications to Restricted Isometry and Beyond
This work addresses a limitation for researchers and practitioners in machine learning and randomized algorithms by enabling the application of foundational results to dependent or adaptive modeling settings, though it is incremental in extending existing theory.
The paper tackles the problem of extending uniform bounds on quadratic forms of random vectors/matrices, which underpin key results like the Johnson-Lindenstrauss Lemma and Restricted Isometry Property, to settings with statistical dependence, showing these results hold under general dependence structures such as conditional independence and sub-Gaussianity given a latent process.
Several important families of computational and statistical results in machine learning and randomized algorithms rely on uniform bounds on quadratic forms of random vectors or matrices. Such results include the Johnson-Lindenstrauss (J-L) Lemma, the Restricted Isometry Property (RIP), randomized sketching algorithms, and approximate linear algebra. The existing results critically depend on statistical independence, e.g., independent entries for random vectors, independent rows for random matrices, etc., which prevent their usage in dependent or adaptive modeling settings. In this paper, we show that such independence is in fact not needed for such results which continue to hold under fairly general dependence structures. In particular, we present uniform bounds on random quadratic forms of stochastic processes which are conditionally independent and sub-Gaussian given another (latent) process. Our setup allows general dependencies of the stochastic process on the history of the latent process and the latent process to be influenced by realizations of the stochastic process. The results are thus applicable to adaptive modeling settings and also allows for sequential design of random vectors and matrices. We also discuss stochastic process based forms of J-L, RIP, and sketching, to illustrate the generality of the results.