An Empirical Study on Disentanglement of Negative-free Contrastive Learning
This work addresses the lack of disentanglement evaluation for high-dimensional representation models in machine learning, particularly for researchers in self-supervised learning, but it is incremental as it extends existing disentanglement studies to a new method.
The paper tackles the problem of evaluating disentanglement in negative-free contrastive learning methods, finding that existing metrics fail for high-dimensional representations, and proposes a new metric based on Mutual Information to benchmark these methods on synthetic and real-world datasets, showing they can learn a well-disentangled subset of representation.
Negative-free contrastive learning methods have attracted a lot of attention with simplicity and impressive performances for large-scale pretraining. However, its disentanglement property remains unexplored. In this paper, we examine negative-free contrastive learning methods to study the disentanglement property empirically. We find that existing disentanglement metrics fail to make meaningful measurements for high-dimensional representation models, so we propose a new disentanglement metric based on Mutual Information between latent representations and data factors. With this proposed metric, we benchmark the disentanglement property of negative-free contrastive learning on both popular synthetic datasets and a real-world dataset CelebA. Our study shows that the investigated methods can learn a well-disentangled subset of representation. As far as we know, we are the first to extend the study of disentangled representation learning to high-dimensional representation space and introduce negative-free contrastive learning methods into this area. The source code of this paper is available at \url{https://github.com/noahcao/disentanglement_lib_med}.