Higher Order Reduced Rank Regression
This is an incremental improvement for multi-response regression problems where linear models are insufficient, targeting researchers and practitioners in machine learning and statistics.
The paper tackles the limitation of Reduced Rank Regression (RRR) in capturing nonlinear interactions by introducing Higher Order Reduced Rank Regression (HORRR), which uses tensor representations and Tucker decomposition to model multilinear relationships, and employs Riemannian optimization for solving the problems.
Reduced Rank Regression (RRR) is a widely used method for multi-response regression. However, RRR assumes a linear relationship between features and responses. While linear models are useful and often provide a good approximation, many real-world problems involve more complex relationships that cannot be adequately captured by simple linear interactions. One way to model such relationships is via multilinear transformations. This paper introduces Higher Order Reduced Rank Regression (HORRR), an extension of RRR that leverages multi-linear transformations, and as such is capable of capturing nonlinear interactions in multi-response regression. HORRR employs tensor representations for the coefficients and a Tucker decomposition to impose multilinear rank constraints as regularization akin to the rank constraints in RRR. Encoding these constraints as a manifold allows us to use Riemannian optimization to solve this HORRR problems. We theoretically and empirically analyze the use of Riemannian optimization for solving HORRR problems.