Joker: Joint Optimization Framework for Lightweight Kernel Machines
This addresses the problem of high memory costs in kernel methods for machine learning practitioners, offering an incremental improvement by extending optimization to multiple models beyond kernel ridge regression.
The paper tackles the scalability and memory overhead issues in large-scale kernel methods by proposing Joker, a joint optimization framework for diverse kernel models, which saves up to 90% memory while achieving comparable or better training time and performance than state-of-the-art methods.
Kernel methods are powerful tools for nonlinear learning with well-established theory. The scalability issue has been their long-standing challenge. Despite the existing success, there are two limitations in large-scale kernel methods: (i) The memory overhead is too high for users to afford; (ii) existing efforts mainly focus on kernel ridge regression (KRR), while other models lack study. In this paper, we propose Joker, a joint optimization framework for diverse kernel models, including KRR, logistic regression, and support vector machines. We design a dual block coordinate descent method with trust region (DBCD-TR) and adopt kernel approximation with randomized features, leading to low memory costs and high efficiency in large-scale learning. Experiments show that Joker saves up to 90\% memory but achieves comparable training time and performance (or even better) than the state-of-the-art methods.