Sigma-Delta and Distributed Noise-Shaping Quantization Methods for Random Fourier Features
This work addresses memory efficiency in kernel approximations for machine learning practitioners, offering an incremental improvement over existing quantization methods.
The paper tackles the problem of quantizing Random Fourier Features (RFFs) for shift-invariant kernels using low bit-depth Sigma-Delta and distributed noise-shaping methods, achieving high accuracy approximations with polynomial error decay as dimension increases and exponential error decay with bits used, as validated empirically on machine learning tasks.
We propose the use of low bit-depth Sigma-Delta and distributed noise-shaping methods for quantizing the Random Fourier features (RFFs) associated with shift-invariant kernels. We prove that our quantized RFFs -- even in the case of $1$-bit quantization -- allow a high accuracy approximation of the underlying kernels, and the approximation error decays at least polynomially fast as the dimension of the RFFs increases. We also show that the quantized RFFs can be further compressed, yielding an excellent trade-off between memory use and accuracy. Namely, the approximation error now decays exponentially as a function of the bits used. Moreover, we empirically show by testing the performance of our methods on several machine learning tasks that our method compares favorably to other state of the art quantization methods in this context.