Adaptive Reduced Rank Regression
This addresses a bottleneck in high-dimensional statistics for researchers and practitioners dealing with limited data, though it appears incremental as it builds on existing low rank regression methods.
The paper tackles the low rank regression problem in extreme high-dimensional settings where the number of observations is less than the sum of feature and response dimensions, by developing an efficient algorithm involving two SVDs that decouples the problem through precision matrix estimation and matrix denoising, with preliminary experiments showing it often outperforms or is competitive with existing baselines.
We study the low rank regression problem $\my = M\mx + ε$, where $\mx$ and $\my$ are $d_1$ and $d_2$ dimensional vectors respectively. We consider the extreme high-dimensional setting where the number of observations $n$ is less than $d_1 + d_2$. Existing algorithms are designed for settings where $n$ is typically as large as $\Rank(M)(d_1+d_2)$. This work provides an efficient algorithm which only involves two SVD, and establishes statistical guarantees on its performance. The algorithm decouples the problem by first estimating the precision matrix of the features, and then solving the matrix denoising problem. To complement the upper bound, we introduce new techniques for establishing lower bounds on the performance of any algorithm for this problem. Our preliminary experiments confirm that our algorithm often out-performs existing baselines, and is always at least competitive.