Automatic Shortcut Removal for Self-Supervised Representation Learning
This addresses a central challenge in self-supervised learning for computer vision by automating shortcut removal, potentially reducing the need for manual design and improving semantic feature extraction, though it is incremental as it builds on existing pretext task frameworks.
The paper tackles the problem of shortcut features in self-supervised visual representation learning, where feature extractors exploit low-level cues like color aberrations, and proposes an automatic framework using an adversarial 'lens' network to remove these shortcuts, resulting in improved representations that outperform baseline methods in all tested cases.
In self-supervised visual representation learning, a feature extractor is trained on a "pretext task" for which labels can be generated cheaply, without human annotation. A central challenge in this approach is that the feature extractor quickly learns to exploit low-level visual features such as color aberrations or watermarks and then fails to learn useful semantic representations. Much work has gone into identifying such "shortcut" features and hand-designing schemes to reduce their effect. Here, we propose a general framework for mitigating the effect shortcut features. Our key assumption is that those features which are the first to be exploited for solving the pretext task may also be the most vulnerable to an adversary trained to make the task harder. We show that this assumption holds across common pretext tasks and datasets by training a "lens" network to make small image changes that maximally reduce performance in the pretext task. Representations learned with the modified images outperform those learned without in all tested cases. Additionally, the modifications made by the lens reveal how the choice of pretext task and dataset affects the features learned by self-supervision.