Semi-supervised learning objectives as log-likelihoods in a generative model of data curation
This provides a theoretical foundation for SSL, potentially enabling Bayesian SSL and explaining why SSL works better on curated data, though it is incremental as it builds on prior work on data curation models.
The paper tackled the lack of a principled understanding of semi-supervised learning (SSL) objectives by formulating them as log-likelihoods in a generative model of data curation, showing that SSL methods like entropy minimization and pseudo-labeling are lower-bounds on this likelihood and providing a proof-of-principle for Bayesian SSL on toy data. It found that SSL gave much larger performance improvements on curated datasets (e.g., Galaxy Zoo 2) than on uncurated ones, explaining past results about SSL effectiveness on clean data.
We currently do not have an understanding of semi-supervised learning (SSL) objectives such as pseudo-labelling and entropy minimization as log-likelihoods, which precludes the development of e.g. Bayesian SSL. Here, we note that benchmark image datasets such as CIFAR-10 are carefully curated, and we formulate SSL objectives as a log-likelihood in a generative model of data curation that was initially developed to explain the cold-posterior effect (Aitchison 2020). SSL methods, from entropy minimization and pseudo-labelling, to state-of-the-art techniques similar to FixMatch can be understood as lower-bounds on our principled log-likelihood. We are thus able to give a proof-of-principle for Bayesian SSL on toy data. Finally, our theory suggests that SSL is effective in part due to the statistical patterns induced by data curation. This provides an explanation of past results which show SSL performs better on clean datasets without any "out of distribution" examples. Confirming these results we find that SSL gave much larger performance improvements on curated than on uncurated data, using matched curated and uncurated datasets based on Galaxy Zoo 2.