Smooth Neighbors on Teacher Graphs for Semi-supervised Learning
This addresses the challenge of limited labeled data in semi-supervised learning for computer vision tasks, offering significant improvements with fewer labels.
The paper tackles the problem of semi-supervised learning by proposing Smooth Neighbors on Teacher Graphs (SNTG), which constructs a graph based on teacher model predictions to smooth representations of similar points, achieving state-of-the-art results with error rates of 9.89% on CIFAR-10 with 4000 labels, 3.99% on SVHN with 500 labels, and reducing error from 4.81% to 1.36% on MNIST with 20 labels.
The recently proposed self-ensembling methods have achieved promising results in deep semi-supervised learning, which penalize inconsistent predictions of unlabeled data under different perturbations. However, they only consider adding perturbations to each single data point, while ignoring the connections between data samples. In this paper, we propose a novel method, called Smooth Neighbors on Teacher Graphs (SNTG). In SNTG, a graph is constructed based on the predictions of the teacher model, i.e., the implicit self-ensemble of models. Then the graph serves as a similarity measure with respect to which the representations of "similar" neighboring points are learned to be smooth on the low-dimensional manifold. We achieve state-of-the-art results on semi-supervised learning benchmarks. The error rates are 9.89%, 3.99% for CIFAR-10 with 4000 labels, SVHN with 500 labels, respectively. In particular, the improvements are significant when the labels are fewer. For the non-augmented MNIST with only 20 labels, the error rate is reduced from previous 4.81% to 1.36%. Our method also shows robustness to noisy labels.