Identifying and Disentangling Spurious Features in Pretrained Image Representations
This work addresses the issue of spurious features in pretrained representations for image classification, which is an incremental improvement over existing methods for mitigating spurious correlations.
The paper tackled the problem of neural networks using spurious correlations in predictions, which reduces performance when these correlations fail, by investigating how spurious features are represented in pretrained image representations and proposing methods to remove them. The result was a significant improvement in classification performance, measured by worst group accuracy, using a linear autoencoder to disentangle and remove spurious features.
Neural networks employ spurious correlations in their predictions, resulting in decreased performance when these correlations do not hold. Recent works suggest fixing pretrained representations and training a classification head that does not use spurious features. We investigate how spurious features are represented in pretrained representations and explore strategies for removing information about spurious features. Considering the Waterbirds dataset and a few pretrained representations, we find that even with full knowledge of spurious features, their removal is not straightforward due to entangled representation. To address this, we propose a linear autoencoder training method to separate the representation into core, spurious, and other features. We propose two effective spurious feature removal approaches that are applied to the encoding and significantly improve classification performance measured by worst group accuracy.