Leveraging Superfluous Information in Contrastive Representation Learning
This work addresses a key bottleneck in self-supervised learning for computer vision by mitigating performance degradation from irrelevant information, though it is incremental as it builds on existing contrastive learning frameworks.
The paper tackles the problem of superfluous task-irrelevant information in contrastive representation learning, which can degrade downstream performance, and introduces a new objective called SuperInfo that combines predictive and superfluous information to learn robust representations, achieving significant improvements in image classification, object detection, and instance segmentation tasks.
Contrastive representation learning, which aims to learnthe shared information between different views of unlabeled data by maximizing the mutual information between them, has shown its powerful competence in self-supervised learning for downstream tasks. However, recent works have demonstrated that more estimated mutual information does not guarantee better performance in different downstream tasks. Such works inspire us to conjecture that the learned representations not only maintain task-relevant information from unlabeled data but also carry task-irrelevant information which is superfluous for downstream tasks, thus leading to performance degeneration. In this paper we show that superfluous information does exist during the conventional contrastive learning framework, and further design a new objective, namely SuperInfo, to learn robust representations by a linear combination of both predictive and superfluous information. Besides, we notice that it is feasible to tune the coefficients of introduced losses to discard task-irrelevant information, while keeping partial non-shared task-relevant information according to our SuperInfo loss.We demonstrate that learning with our loss can often outperform the traditional contrastive learning approaches on image classification, object detection and instance segmentation tasks with significant improvements.