GPS-SSL: Guided Positive Sampling to Inject Prior Into Self-Supervised Learning
This work addresses the challenge for machine learning researchers and practitioners of making SSL less dependent on tailored data augmentations, offering a general method applicable across domains, though it appears incremental as it builds on existing SSL frameworks like SimCLR or BYOL.
The paper tackles the problem of self-supervised learning (SSL) being overly reliant on data augmentations (DA) for generating positive samples by proposing GPS-SSL, a method to inject prior knowledge into SSL through guided positive sampling in a metric space, resulting in improved performance such as 85.58% accuracy on Cifar10 with weak DA compared to 37.51% for the baseline.
We propose Guided Positive Sampling Self-Supervised Learning (GPS-SSL), a general method to inject a priori knowledge into Self-Supervised Learning (SSL) positive samples selection. Current SSL methods leverage Data-Augmentations (DA) for generating positive samples and incorporate prior knowledge - an incorrect, or too weak DA will drastically reduce the quality of the learned representation. GPS-SSL proposes instead to design a metric space where Euclidean distances become a meaningful proxy for semantic relationship. In that space, it is now possible to generate positive samples from nearest neighbor sampling. Any prior knowledge can now be embedded into that metric space independently from the employed DA. From its simplicity, GPS-SSL is applicable to any SSL method, e.g. SimCLR or BYOL. A key benefit of GPS-SSL is in reducing the pressure in tailoring strong DAs. For example GPS-SSL reaches 85.58% on Cifar10 with weak DA while the baseline only reaches 37.51%. We therefore move a step forward towards the goal of making SSL less reliant on DA. We also show that even when using strong DAs, GPS-SSL outperforms the baselines on under-studied domains. We evaluate GPS-SSL along with multiple baseline SSL methods on numerous downstream datasets from different domains when the models use strong or minimal data augmentations. We hope that GPS-SSL will open new avenues in studying how to inject a priori knowledge into SSL in a principled manner.