Towards Understanding Why FixMatch Generalizes Better Than Supervised Learning
This work addresses a theoretical gap in understanding SSL generalization for researchers and practitioners in machine learning, offering insights that could improve algorithm design, though it is incremental as it builds on existing FixMatch methods.
The paper tackles the problem of why FixMatch-like semi-supervised learning (SSL) generalizes better than supervised learning (SL) on deep neural networks, providing the first theoretical justification by analyzing convolutional neural networks on classification tasks, and it develops an improved variant, Semantic-Aware FixMatch, which shows enhanced generalization in experiments.
Semi-supervised learning (SSL), exemplified by FixMatch (Sohn et al., 2020), has shown significant generalization advantages over supervised learning (SL), particularly in the context of deep neural networks (DNNs). However, it is still unclear, from a theoretical standpoint, why FixMatch-like SSL algorithms generalize better than SL on DNNs. In this work, we present the first theoretical justification for the enhanced test accuracy observed in FixMatch-like SSL applied to DNNs by taking convolutional neural networks (CNNs) on classification tasks as an example. Our theoretical analysis reveals that the semantic feature learning processes in FixMatch and SL are rather different. In particular, FixMatch learns all the discriminative features of each semantic class, while SL only randomly captures a subset of features due to the well-known lottery ticket hypothesis. Furthermore, we show that our analysis framework can be applied to other FixMatch-like SSL methods, e.g., FlexMatch, FreeMatch, Dash, and SoftMatch. Inspired by our theoretical analysis, we develop an improved variant of FixMatch, termed Semantic-Aware FixMatch (SA-FixMatch). Experimental results corroborate our theoretical findings and the enhanced generalization capability of SA-FixMatch.