Tensor-Train Networks for Learning Predictive Modeling of Multidimensional Data
This work addresses the challenge of high computational and memory costs in neural networks for researchers and practitioners, though it is incremental as it builds on existing tensor decomposition methods.
The paper tackles the problem of reducing the parameter count in neural networks by applying Tensor-Train (TT) networks to compress Multilayer Perceptrons, achieving up to 95% coefficient reduction. It demonstrates improved robustness in optimization and faster convergence using an alternating least squares algorithm on tasks like chaotic time series and stock index prediction.
In this work, we firstly apply the Train-Tensor (TT) networks to construct a compact representation of the classical Multilayer Perceptron, representing a reduction of up to 95% of the coefficients. A comparative analysis between tensor model and standard multilayer neural networks is also carried out in the context of prediction of the Mackey-Glass noisy chaotic time series and NASDAQ index. We show that the weights of a multidimensional regression model can be learned by means of TT network and the optimization of TT weights is a more robust to the impact of coefficient initialization and hyper-parameter setting. Furthermore, an efficient algorithm based on alternating least squares has been proposed for approximating the weights in TT-format with a reduction of computational calculus, providing a much faster convergence than the well-known adaptive learning-method algorithms, widely applied for optimizing neural networks.