Rademacher Random Projections with Tensor Networks
This work offers an incremental improvement for machine learning practitioners dealing with high-dimensional data compression, though it is domain-specific to tensor-based methods.
The paper tackles the problem of high-dimensional tensor dimension reduction by proposing a tensorized random projection using Rademacher distributions in Tensor Train format, showing it can outperform Gaussian-based methods in experiments on synthetic data.
Random projection (RP) have recently emerged as popular techniques in the machine learning community for their ability in reducing the dimension of very high-dimensional tensors. Following the work in [30], we consider a tensorized random projection relying on Tensor Train (TT) decomposition where each element of the core tensors is drawn from a Rademacher distribution. Our theoretical results reveal that the Gaussian low-rank tensor represented in compressed form in TT format in [30] can be replaced by a TT tensor with core elements drawn from a Rademacher distribution with the same embedding size. Experiments on synthetic data demonstrate that tensorized Rademacher RP can outperform the tensorized Gaussian RP studied in [30]. In addition, we show both theoretically and experimentally, that the tensorized RP in the Matrix Product Operator (MPO) format is not a Johnson-Lindenstrauss transform (JLT) and therefore not a well-suited random projection map