Quantized Fourier and Polynomial Features for more Expressive Tensor Network Models
This work addresses the computational bottleneck in high-dimensional machine learning for researchers and practitioners, offering a method to enhance model expressiveness without extra cost, though it is incremental as it builds on existing tensor network approaches.
The paper tackles the exponential scaling of model parameters in high-dimensional kernel machines by quantizing polynomial and Fourier features and their associated weights, resulting in quantized models with higher VC-dimension bounds and improved generalization, achieving state-of-the-art results on a large regression task on a laptop computer.
In the context of kernel machines, polynomial and Fourier features are commonly used to provide a nonlinear extension to linear models by mapping the data to a higher-dimensional space. Unless one considers the dual formulation of the learning problem, which renders exact large-scale learning unfeasible, the exponential increase of model parameters in the dimensionality of the data caused by their tensor-product structure prohibits to tackle high-dimensional problems. One of the possible approaches to circumvent this exponential scaling is to exploit the tensor structure present in the features by constraining the model weights to be an underparametrized tensor network. In this paper we quantize, i.e. further tensorize, polynomial and Fourier features. Based on this feature quantization we propose to quantize the associated model weights, yielding quantized models. We show that, for the same number of model parameters, the resulting quantized models have a higher bound on the VC-dimension as opposed to their non-quantized counterparts, at no additional computational cost while learning from identical features. We verify experimentally how this additional tensorization regularizes the learning problem by prioritizing the most salient features in the data and how it provides models with increased generalization capabilities. We finally benchmark our approach on large regression task, achieving state-of-the-art results on a laptop computer.