PAC-Bayesian Contrastive Unsupervised Representation Learning
This work provides theoretical guarantees for CURL, addressing a gap for researchers in machine learning, though it is incremental as it builds on existing frameworks.
The paper tackles the lack of theoretical understanding in contrastive unsupervised representation learning (CURL) by extending prior work to the PAC-Bayes setting, enabling non-iid analysis and deriving a new algorithm that achieves competitive accuracy and non-vacuous generalization bounds on real-life datasets.
Contrastive unsupervised representation learning (CURL) is the state-of-the-art technique to learn representations (as a set of features) from unlabelled data. While CURL has collected several empirical successes recently, theoretical understanding of its performance was still missing. In a recent work, Arora et al. (2019) provide the first generalisation bounds for CURL, relying on a Rademacher complexity. We extend their framework to the flexible PAC-Bayes setting, allowing us to deal with the non-iid setting. We present PAC-Bayesian generalisation bounds for CURL, which are then used to derive a new representation learning algorithm. Numerical experiments on real-life datasets illustrate that our algorithm achieves competitive accuracy, and yields non-vacuous generalisation bounds.