Linearly Convergent Mixup Learning
This work addresses a specific bottleneck in RKHS-based learning for binary classification, offering incremental improvements in efficiency and performance for scenarios with limited training data.
The paper tackles the challenge of applying mixup data augmentation to learning in reproducing kernel Hilbert spaces (RKHS) for binary classification, where intermediate class labels pose difficulties for dual optimization methods, and presents two novel algorithms that achieve faster convergence to the optimal solution compared to gradient descent approaches, with mixup consistently improving predictive performance across various loss functions.
Learning in the reproducing kernel Hilbert space (RKHS) such as the support vector machine has been recognized as a promising technique. It continues to be highly effective and competitive in numerous prediction tasks, particularly in settings where there is a shortage of training data or computational limitations exist. These methods are especially valued for their ability to work with small datasets and their interpretability. To address the issue of limited training data, mixup data augmentation, widely used in deep learning, has remained challenging to apply to learning in RKHS due to the generation of intermediate class labels. Although gradient descent methods handle these labels effectively, dual optimization approaches are typically not directly applicable. In this study, we present two novel algorithms that extend to a broader range of binary classification models. Unlike gradient-based approaches, our algorithms do not require hyperparameters like learning rates, simplifying their implementation and optimization. Both the number of iterations to converge and the computational cost per iteration scale linearly with respect to the dataset size. The numerical experiments demonstrate that our algorithms achieve faster convergence to the optimal solution compared to gradient descent approaches, and that mixup data augmentation consistently improves the predictive performance across various loss functions.