Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
This addresses a theoretical gap for researchers in self-supervised learning, though it appears incremental as it clarifies an existing inconsistency without introducing a new method.
The paper tackles the inconsistency in instance discriminative self-supervised learning where theory suggests many negative samples degrade performance, but empirical results show improvement. It provides a novel analysis using the coupon collector's problem to explain this, confirming the findings on real-world datasets.
Instance discriminative self-supervised representation learning has been attracted attention thanks to its unsupervised nature and informative feature representation for downstream tasks. In practice, it commonly uses a larger number of negative samples than the number of supervised classes. However, there is an inconsistency in the existing analysis; theoretically, a large number of negative samples degrade classification performance on a downstream supervised task, while empirically, they improve the performance. We provide a novel framework to analyze this empirical result regarding negative samples using the coupon collector's problem. Our bound can implicitly incorporate the supervised loss of the downstream task in the self-supervised loss by increasing the number of negative samples. We confirm that our proposed analysis holds on real-world benchmark datasets.