Convex Loss Functions for Support Vector Machines (SVMs) and Neural Networks
This is an incremental improvement for machine learning practitioners seeking better performance in classification and regression tasks.
The paper tackles the problem of improving generalization in Support Vector Machines and neural networks by proposing a new convex loss function that incorporates pattern correlations, resulting in up to 2.0% higher F1 scores for classification and 1.0% lower MSE for regression compared to standard losses.
We propose a new convex loss for Support Vector Machines, both for the binary classification and for the regression models. Therefore, we show the mathematical derivation of the dual problems and we experiment with them on several small datasets. The minimal dimension of those datasets is due to the difficult scalability of the SVM method to bigger instances. This preliminary study should prove that using pattern correlations inside the loss function could enhance the generalisation performances. Our method consistently achieved comparable or superior performance, with improvements of up to 2.0% in F1 scores for classification tasks and 1.0% reduction in Mean Squared Error (MSE) for regression tasks across various datasets, compared to standard losses. Coherently, results show that generalisation measures are never worse than the standard losses and several times they are better. In our opinion, it should be considered a careful study of this loss, coupled with shallow and deep neural networks. In fact, we present some novel results obtained with those architectures.