Regression-aware decompositions
This provides a supervised approach to dimensionality reduction, which is incremental as it builds on existing unsupervised methods by integrating regression information.
The paper tackles the problem of making classical unsupervised dimensionality reduction methods like PCA and interpolative decomposition supervised by incorporating linear least-squares regression with a design matrix A, resulting in regression-aware decompositions that reveal structure in B relevant to regression against A.
Linear least-squares regression with a "design" matrix A approximates a given matrix B via minimization of the spectral- or Frobenius-norm discrepancy ||AX-B|| over every conformingly sized matrix X. Another popular approximation is low-rank approximation via principal component analysis (PCA) -- which is essentially singular value decomposition (SVD) -- or interpolative decomposition (ID). Classically, PCA/SVD and ID operate solely with the matrix B being approximated, not supervised by any auxiliary matrix A. However, linear least-squares regression models can inform the ID, yielding regression-aware ID. As a bonus, this provides an interpretation as regression-aware PCA for a kind of canonical correlation analysis between A and B. The regression-aware decompositions effectively enable supervision to inform classical dimensionality reduction, which classically has been totally unsupervised. The regression-aware decompositions reveal the structure inherent in B that is relevant to regression against A.