Self-Supervision on Images and Text Reduces Reliance on Visual Shortcut Features
This addresses the issue of shortcut features causing failures in real-world applications like medical imaging, though it is an incremental improvement rather than a complete solution.
The paper tackled the problem of deep learning models relying on visual shortcut features that degrade performance in deployment, finding that self-supervised models trained on images and text reduce reliance on these features and forget them more quickly during fine-tuning, with experiments showing improved robustness in a medical imaging example.
Deep learning models trained in a fully supervised manner have been shown to rely on so-called "shortcut" features. Shortcut features are inputs that are associated with the outcome of interest in the training data, but are either no longer associated or not present in testing or deployment settings. Here we provide experiments that show recent self-supervised models trained on images and text provide more robust image representations and reduce the model's reliance on visual shortcut features on a realistic medical imaging example. Additionally, we find that these self-supervised models "forget" shortcut features more quickly than fully supervised ones when fine-tuned on labeled data. Though not a complete solution, our experiments provide compelling evidence that self-supervised models trained on images and text provide some resilience to visual shortcut features.