Laplace Approximation For Tensor Train Kernel Machines In System Identification
This work addresses scalability issues in Gaussian process regression for system identification, offering a more efficient probabilistic method for practitioners in that domain.
The paper tackles the challenge of making tensor train kernel machines fully probabilistic by introducing a Bayesian approach that uses Laplace approximation for a selected TT-core and variational inference for hyperparameters, achieving up to 65x faster training compared to cross-validation.
To address the scalability limitations of Gaussian process (GP) regression, several approximation techniques have been proposed. One such method is based on tensor networks, which utilizes an exponential number of basis functions without incurring exponential computational cost. However, extending this model to a fully probabilistic formulation introduces several design challenges. In particular, for tensor train (TT) models, it is unclear which TT-core should be treated in a Bayesian manner. We introduce a Bayesian tensor train kernel machine that applies Laplace approximation to estimate the posterior distribution over a selected TT-core and employs variational inference (VI) for precision hyperparameters. Experiments show that core selection is largely independent of TT-ranks and feature structure, and that VI replaces cross-validation while offering up to 65x faster training. The method's effectiveness is demonstrated on an inverse dynamics problem.