Bringing the Context Back into Object Recognition, Robustly
This addresses robustness issues in object recognition for real-world deployment, though it is incremental as it builds on existing localization techniques.
The paper tackles the problem of object recognition models over-relying on background context, which limits robustness to distribution shifts, by proposing L2R2, a method that localizes foreground objects before recognition. It improves performance on standard and zero-shot recognition tasks, showing robustness to long-tail backgrounds and distribution shifts across various datasets.
In object recognition, both the subject of interest (referred to as foreground, FG, for simplicity) and its surrounding context (background, BG) may play an important role. However, standard supervised learning often leads to unintended over-reliance on the BG, limiting model robustness in real-world deployment settings. The problem is mainly addressed by suppressing the BG, sacrificing context information for improved generalization. We propose "Localize to Recognize Robustly" (L2R2), a novel recognition approach which exploits the benefits of context-aware classification while maintaining robustness to distribution shifts. L2R2 leverages advances in zero-shot detection to localize the FG before recognition. It improves the performance of both standard recognition with supervised training, as well as multimodal zero-shot recognition with VLMs, while being robust to long-tail BGs and distribution shifts. The results confirm localization before recognition is possible for a wide range of datasets and they highlight the limits of object detection on others