On the Complexity of Learning with Kernels
This work addresses computational efficiency for researchers and practitioners in machine learning, but it is incremental as it builds on existing methods to provide theoretical insights.
The paper tackles the computational bottleneck in kernel learning by analyzing lower bounds on error for methods that reduce kernel matrix size, showing that some problems do not allow non-trivial savings and quantifying dependencies on parameters like loss function and kernel rank.
A well-recognized limitation of kernel learning is the requirement to handle a kernel matrix, whose size is quadratic in the number of training examples. Many methods have been proposed to reduce this computational cost, mostly by using a subset of the kernel matrix entries, or some form of low-rank matrix approximation, or a random projection method. In this paper, we study lower bounds on the error attainable by such methods as a function of the number of entries observed in the kernel matrix or the rank of an approximate kernel matrix. We show that there are kernel learning problems where no such method will lead to non-trivial computational savings. Our results also quantify how the problem difficulty depends on parameters such as the nature of the loss function, the regularization parameter, the norm of the desired predictor, and the kernel matrix rank. Our results also suggest cases where more efficient kernel learning might be possible.