Maximally Useful and Minimally Redundant: The Key to Self Supervised Learning for Imbalanced Data
This addresses dataset imbalance in self-supervised learning, which is a domain-specific problem for machine learning practitioners working with skewed data distributions.
The paper tackles the problem of contrastive self-supervised learning (CSSL) not generalizing well to imbalanced datasets by proposing a method based on mutual information and more than two views to extract better representations for tail classes. It achieves state-of-the-art accuracy improvements, such as 2% on Cifar10-LT and 5% on Cifar100-LT using Resnet-18.
The robustness of contrastive self-supervised learning (CSSL) for imbalanced datasets is largely unexplored. CSSL usually makes use of \emph{multi-view} assumptions to learn discriminatory features via similar and dissimilar data samples. CSSL works well on balanced datasets, but does not generalize well for imbalanced datasets. In a very recent paper, as part of future work, Yann LeCun pointed out that the self-supervised multiview framework can be extended to cases involving \emph{more than two views}. Taking a cue from this insight we propose a theoretical justification based on the concept of \emph{mutual information} to support the \emph{more than two views} objective and apply it to the problem of dataset imbalance in self-supervised learning. The proposed method helps extract representative characteristics of the tail classes by segregating between \emph{intra} and \emph{inter} discriminatory characteristics. We introduce a loss function that helps us to learn better representations by filtering out extreme features. Experimental evaluation on a variety of self-supervised frameworks (both contrastive and non-contrastive) also prove that the \emph{more than two view} objective works well for imbalanced datasets. We achieve a new state-of-the-art accuracy in self-supervised imbalanced dataset classification (2\% improvement in Cifar10-LT using Resnet-18, 5\% improvement in Cifar100-LT using Resnet-18, 3\% improvement in Imagenet-LT (1k) using Resnet-50).