Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualization
This addresses a bottleneck in self-supervised learning for handling imbalanced datasets, offering a methodical approach to improve feature separability and robustness.
The paper tackles the problem of using a single global temperature parameter in contrastive self-supervised learning, which fails to account for varying semantic frequencies in data, especially in long-tailed distributions, and proposes an automatic temperature individualization method that outperforms strong baselines like SimCLR and CLIP, with larger improvements on imbalanced data.
In this paper, we aim to optimize a contrastive loss with individualized temperatures in a principled and systematic manner for self-supervised learning. The common practice of using a global temperature parameter $τ$ ignores the fact that ``not all semantics are created equal", meaning that different anchor data may have different numbers of samples with similar semantics, especially when data exhibits long-tails. First, we propose a new robust contrastive loss inspired by distributionally robust optimization (DRO), providing us an intuition about the effect of $τ$ and a mechanism for automatic temperature individualization. Then, we propose an efficient stochastic algorithm for optimizing the robust contrastive loss with a provable convergence guarantee without using large mini-batch sizes. Theoretical and experimental results show that our algorithm automatically learns a suitable $τ$ for each sample. Specifically, samples with frequent semantics use large temperatures to keep local semantic structures, while samples with rare semantics use small temperatures to induce more separable features. Our method not only outperforms prior strong baselines (e.g., SimCLR, CLIP) on unimodal and bimodal datasets with larger improvements on imbalanced data but also is less sensitive to hyper-parameters. To our best knowledge, this is the first methodical approach to optimizing a contrastive loss with individualized temperatures.