On the Curse of Dimensionality in Private Sparse Covariance Estimation and PCA
For researchers in private high-dimensional statistics, this work reveals a fundamental curse of dimensionality in sparse covariance estimation under differential privacy, showing that sparsity alone does not alleviate the need for many samples.
The paper studies differentially private sparse covariance estimation and PCA under row-column sparsity, proving an exponential gap between private and non-private sample complexity when sparsity is polylogarithmic. It shows that poly(k, log d) samples suffice for private PCA with sparse leading eigenvector, but poly(d) samples are necessary for both tasks under DP, establishing the first such separation for sparse estimation.
We study high-dimensional differentially private (DP) covariance estimation in the operator norm, and principal component analysis (PCA), under $k$-row-column sparsity ($k$-RCS) of the covariance matrix. In the non-private setting, it is known that $\mathsf{poly}(k, \log d)$ samples suffice to solve both of these problems. However, the only comparable result known under DP (Wang et al. 2021) requires $Ω(d)$ samples under standard parameterizations of the problem. We investigate when this curse of dimensionality is inherent for sparse covariance estimation tasks under DP. On the upper bound front, we show that a $\mathsf{poly}(k, \log d)$ sample complexity for PCA is possible under DP, if we also posit sparsity of the leading eigenvector. We complement this result with $\mathsf{poly}(d)$ lower bounds under DP for both sparse covariance estimation and PCA, establishing an exponential gap between the private and non-private variants of these problems when $k = \mathsf{polylog}(d)$. To our knowledge, no such separation has previously been demonstrated for any sparse estimation problems in private high-dimensional statistics. Our techniques are flexible enough that they imply stronger lower bounds even for the well-studied problem of standard DP PCA, without sparsity assumptions.