Natural Riemannian gradient for learning functional tensor networks
This work addresses the challenge of training functional tensor networks for machine learning tasks beyond least-squares regression, offering a more efficient optimization method for researchers and practitioners in computational mathematics and machine learning, though it is incremental as it builds on existing natural gradient concepts.
The authors tackled the problem of efficiently optimizing low-rank functional tree tensor networks (TTNs) for arbitrary loss functions, such as multinomial logistic regression, by proposing a natural Riemannian gradient descent method. The result showed that this approach significantly improved convergence behavior compared to standard Riemannian gradient methods, as confirmed by numerical experiments on classification datasets.
We consider machine learning tasks with low-rank functional tree tensor networks (TTN) as the learning model. While in the case of least-squares regression, low-rank functional TTNs can be efficiently optimized using alternating optimization, this is not directly possible in other problems, such as multinomial logistic regression. We propose a natural Riemannian gradient descent type approach applicable to arbitrary losses which is based on the natural gradient by Amari. In particular, the search direction obtained by the natural gradient is independent of the choice of basis of the underlying functional tensor product space. Our framework applies to both the factorized and manifold-based approach for representing the functional TTN. For practical application, we propose a hierarchy of efficient approximations to the true natural Riemannian gradient for computing the updates in the parameter space. Numerical experiments confirm our theoretical findings on common classification datasets and show that using natural Riemannian gradient descent for learning considerably improves convergence behavior when compared to standard Riemannian gradient methods.