Learning Bounds for Greedy Approximation with Explicit Feature Maps from Multiple Kernels
This work addresses the challenge of kernel approximation for efficient machine learning, offering an incremental improvement over existing data-dependent random feature methods.
The paper tackles the problem of efficiently approximating nonlinear kernels with explicit feature maps by greedily selecting features from multiple kernels, establishing an error bound that shows test error can be controlled with a small number of features when the underlying model is sparse, and empirically achieving lower test error and time cost compared to state-of-the-art methods.
Nonlinear kernels can be approximated using finite-dimensional feature maps for efficient risk minimization. Due to the inherent trade-off between the dimension of the (mapped) feature space and the approximation accuracy, the key problem is to identify promising (explicit) features leading to a satisfactory out-of-sample performance. In this work, we tackle this problem by efficiently choosing such features from multiple kernels in a greedy fashion. Our method sequentially selects these explicit features from a set of candidate features using a correlation metric. We establish an out-of-sample error bound capturing the trade-off between the error in terms of explicit features (approximation error) and the error due to spectral properties of the best model in the Hilbert space associated to the combined kernel (spectral error). The result verifies that when the (best) underlying data model is sparse enough, i.e., the spectral error is negligible, one can control the test error with a small number of explicit features, that can scale poly-logarithmically with data. Our empirical results show that given a fixed number of explicit features, the method can achieve a lower test error with a smaller time cost, compared to the state-of-the-art in data-dependent random features.