Spuriosity Rankings for Free: A Simple Framework for Last Layer Retraining Based on Object Detection
This addresses reliability issues in deep learning by automating the selection of data subsets for retraining, though it is incremental as it builds on existing last-layer retraining methods.
The paper tackles the problem of deep neural networks relying on spurious features by proposing a ranking framework that uses open vocabulary object detection to identify images without spurious cues for last-layer retraining, achieving effectiveness on the ImageNet-1k dataset.
Deep neural networks have exhibited remarkable performance in various domains. However, the reliance of these models on spurious features has raised concerns about their reliability. A promising solution to this problem is last-layer retraining, which involves retraining the linear classifier head on a small subset of data without spurious cues. Nevertheless, selecting this subset requires human supervision, which reduces its scalability. Moreover, spurious cues may still exist in the selected subset. As a solution to this problem, we propose a novel ranking framework that leverages an open vocabulary object detection technique to identify images without spurious cues. More specifically, we use the object detector as a measure to score the presence of the target object in the images. Next, the images are sorted based on this score, and the last-layer of the model is retrained on a subset of the data with the highest scores. Our experiments on the ImageNet-1k dataset demonstrate the effectiveness of this ranking framework in sorting images based on spuriousness and using them for last-layer retraining.