Rethinking The Uniformity Metric in Self-Supervised Learning
This work addresses a specific evaluation problem in self-supervised learning, offering an incremental improvement for researchers and practitioners in representation learning.
The paper identifies limitations in an existing uniformity metric for self-supervised learning and proposes a new metric based on the Wasserstein distance that satisfies key properties, leading to improved performance on CIFAR-10 and CIFAR-100 datasets.
Uniformity plays an important role in evaluating learned representations, providing insights into self-supervised learning. In our quest for effective uniformity metrics, we pinpoint four principled properties that such metrics should possess. Namely, an effective uniformity metric should remain invariant to instance permutations and sample replications while accurately capturing feature redundancy and dimensional collapse. Surprisingly, we find that the uniformity metric proposed by \citet{Wang2020UnderstandingCR} fails to satisfy the majority of these properties. Specifically, their metric is sensitive to sample replications, and can not account for feature redundancy and dimensional collapse correctly. To overcome these limitations, we introduce a new uniformity metric based on the Wasserstein distance, which satisfies all the aforementioned properties. Integrating this new metric in existing self-supervised learning methods effectively mitigates dimensional collapse and consistently improves their performance on downstream tasks involving CIFAR-10 and CIFAR-100 datasets. Code is available at \url{https://github.com/statsle/WassersteinSSL}.