GEDI: GEnerative and DIscriminative Training for Self-Supervised Learning
This work addresses the challenge of improving self-supervised learning for machine learning practitioners by offering a unified approach, though it appears incremental as it builds on existing methods.
The paper tackled the problem of unifying self-supervised learning objectives by proposing GEDI, a framework that integrates generative and discriminative training, and showed it outperforms existing methods in clustering performance on datasets like SVHN, CIFAR10, and CIFAR100.
Self-supervised learning is a popular and powerful method for utilizing large amounts of unlabeled data, for which a wide variety of training objectives have been proposed in the literature. In this study, we perform a Bayesian analysis of state-of-the-art self-supervised learning objectives and propose a unified formulation based on likelihood learning. Our analysis suggests a simple method for integrating self-supervised learning with generative models, allowing for the joint training of these two seemingly distinct approaches. We refer to this combined framework as GEDI, which stands for GEnerative and DIscriminative training. Additionally, we demonstrate an instantiation of the GEDI framework by integrating an energy-based model with a cluster-based self-supervised learning model. Through experiments on synthetic and real-world data, including SVHN, CIFAR10, and CIFAR100, we show that GEDI outperforms existing self-supervised learning strategies in terms of clustering performance by a wide margin. We also demonstrate that GEDI can be integrated into a neural-symbolic framework to address tasks in the small data regime, where it can use logical constraints to further improve clustering and classification performance.