Data Augmentation of Contrastive Learning is Estimating Positive-incentive Noise
This work addresses improving contrastive learning by theoretically linking it to noise estimation, offering a novel framework for generating data augmentations across diverse data types, though it appears incremental as it builds on existing contrastive models.
The paper investigates the connection between contrastive learning and Positive-incentive Noise (π-Noise), showing that standard data augmentation in contrastive learning can be viewed as an estimation of π-Noise, and proposes a framework to learn beneficial noise as augmentations, with visualization confirming effective augmentations.
Inspired by the idea of Positive-incentive Noise (Pi-Noise or $π$-Noise) that aims at learning the reliable noise beneficial to tasks, we scientifically investigate the connection between contrastive learning and $π$-noise in this paper. By converting the contrastive loss to an auxiliary Gaussian distribution to quantitatively measure the difficulty of the specific contrastive model under the information theory framework, we properly define the task entropy, the core concept of $π$-noise, of contrastive learning. It is further proved that the predefined data augmentation in the standard contrastive learning paradigm can be regarded as a kind of point estimation of $π$-noise. Inspired by the theoretical study, a framework that develops a $π$-noise generator to learn the beneficial noise (instead of estimation) as data augmentations for contrast is proposed. The designed framework can be applied to diverse types of data and is also completely compatible with the existing contrastive models. From the visualization, we surprisingly find that the proposed method successfully learns effective augmentations.