Leveraging Pretrained Image Classifiers for Language-Based Segmentation
This addresses the need for more flexible segmentation models in computer vision, though it is incremental as it builds on existing architectures and priors.
The paper tackles the problem of semantic segmentation models' inability to generalize to unseen object classes without retraining, proposing a model that uses pretrained image classifier activations as visual priors to segment new labels, achieving performance without additional annotated data.
Current semantic segmentation models cannot easily generalize to new object classes unseen during train time: they require additional annotated images and retraining. We propose a novel segmentation model that injects visual priors into semantic segmentation architectures, allowing them to segment out new target labels without retraining. As visual priors, we use the activations of pretrained image classifiers, which provide noisy indications of the spatial location of both the target object and distractor objects in the scene. We leverage language semantics to obtain these activations for a target label unseen by the classifier. Further experiments show that the visual priors obtained via language semantics for both relevant and distracting objects are key to our performance.