Optimal Model Averaging of Support Vector Machines in Diverging Model Spaces
This work provides a novel model averaging approach for SVM, addressing a gap in frequentist methods for practitioners dealing with high-dimensional classification problems.
The authors tackled the problem of model selection for support vector machines (SVM) with high-dimensional covariates by proposing a frequentist model averaging method that selects optimal weights via cross-validation, showing asymptotic optimality with a hinge loss ratio converging to one and avoiding the need for tuning parameter selection.
Support vector machine (SVM) is a powerful classification method that has achieved great success in many fields. Since its performance can be seriously impaired by redundant covariates, model selection techniques are widely used for SVM with high dimensional covariates. As an alternative to model selection, significant progress has been made in the area of model averaging in the past decades. Yet no frequentist model averaging method was considered for SVM. This work aims to fill the gap and to propose a frequentist model averaging procedure for SVM which selects the optimal weight by cross validation. Even when the number of covariates diverges at an exponential rate of the sample size, we show asymptotic optimality of the proposed method in the sense that the ratio of its hinge loss to the lowest possible loss converges to one. We also derive the convergence rate which provides more insights to model averaging. Compared to model selection methods of SVM which require a tedious but critical task of tuning parameter selection, the model averaging method avoids the task and shows promising performances in the empirical studies.