Asymptotic Optimism for Tensor Regression Models with Applications to Neural Network Compression
This provides a theoretical foundation for rank selection in tensor regression, with practical applications in neural network compression, though it appears incremental as it extends existing optimism concepts to tensor models.
The authors tackled the problem of rank selection for low-rank tensor regression models by deriving asymptotic expressions for the expected training-testing discrepancy (optimism) under Gaussian random designs, showing it is minimized at the true tensor rank for both CP and Tucker decompositions, which led to a prediction-oriented rank-selection rule validated on a real-world image regression task and applied to neural network compression.
We study rank selection for low-rank tensor regression under random covariates design. Under a Gaussian random-design model and some mild conditions, we derive population expressions for the expected training-testing discrepancy (optimism) for both CP and Tucker decomposition. We further demonstrate that the optimism is minimized at the true tensor rank for both CP and Tucker regression. This yields a prediction-oriented rank-selection rule that aligns with cross-validation and extends naturally to tensor-model averaging. We also discuss conditions under which under- or over-ranked models may appear preferable, thereby clarifying the scope of the method. Finally, we showcase its practical utility on a real-world image regression task and extend its application to tensor-based compression of neural network, highlighting its potential for model selection in deep learning.