Efficient KLMS and KRLS Algorithms: A Random Fourier Feature Perspective
This work addresses computational bottlenecks for researchers and practitioners using online kernel methods in machine learning, though it is incremental as it builds on existing random feature techniques.
The authors tackled the computational inefficiency of online kernel least squares algorithms by proposing a framework that uses random Fourier features to map data to a fixed-dimensional Euclidean space, resulting in algorithms that are significantly more efficient while maintaining similar convergence speeds and error floors.
We present a new framework for online Least Squares algorithms for nonlinear modeling in RKH spaces (RKHS). Instead of implicitly mapping the data to a RKHS (e.g., kernel trick), we map the data to a finite dimensional Euclidean space, using random features of the kernel's Fourier transform. The advantage is that, the inner product of the mapped data approximates the kernel function. The resulting "linear" algorithm does not require any form of sparsification, since, in contrast to all existing algorithms, the solution's size remains fixed and does not increase with the iteration steps. As a result, the obtained algorithms are computationally significantly more efficient compared to previously derived variants, while, at the same time, they converge at similar speeds and to similar error floors.