Estimating Graph Dimension with Cross-validated Eigenvalues
This addresses a fundamental problem in applied multivariate statistics for estimating latent dimensions in random graphs, offering a method that adapts to cases where some dimensions are not detectable, but it appears incremental as it builds on existing eigenvalue-based approaches.
The paper tackles the problem of estimating the number of latent dimensions in graphs by introducing cross-validated eigenvalues, which provide p-values to test if sample eigenvectors are orthogonal to true dimensions, and shows consistent estimation of k in simulations with favorable computational and statistical performance.
In applied multivariate statistics, estimating the number of latent dimensions or the number of clusters, $k$, is a fundamental and recurring problem. We study a sequence of statistics called "cross-validated eigenvalues." Under a large class of random graph models, including both Poisson and Bernoulli edges, without parametric assumptions, we provide a $p$-value for each cross-validated eigenvalue. It tests the null hypothesis that the sample eigenvector is orthogonal to (i.e., uncorrelated with) the true latent dimensions. This approach naturally adapts to problems where some dimensions are not statistically detectable. In scenarios where all $k$ dimensions can be estimated, we show that our procedure consistently estimates $k$. In simulations and data example, the proposed estimator compares favorably to alternative approaches in both computational and statistical performance.