Investigating the Role of Attribute Context in Vision-Language Models for Object Recognition and Detection
This addresses a gap in understanding attribute utilization in vision-language models for tasks like object detection and fine-grained retrieval, though it is incremental as it builds on existing alignment methods.
The study investigated how vision-language models use attribute context from captions for object recognition and detection, finding that current methods often waste attribute information and fail to adequately incorporate attribute meaning into embeddings, but contrastive training with adjective-based negative captions can improve benefits.
Vision-language alignment learned from image-caption pairs has been shown to benefit tasks like object recognition and detection. Methods are mostly evaluated in terms of how well object class names are learned, but captions also contain rich attribute context that should be considered when learning object alignment. It is unclear how methods use this context in learning, as well as whether models succeed when tasks require attribute and object understanding. To address this gap, we conduct extensive analysis of the role of attributes in vision-language models. We specifically measure model sensitivity to the presence and meaning of attribute context, gauging influence on object embeddings through unsupervised phrase grounding and classification via description methods. We further evaluate the utility of attribute context in training for open-vocabulary object detection, fine-grained text-region retrieval, and attribution tasks. Our results show that attribute context can be wasted when learning alignment for detection, attribute meaning is not adequately considered in embeddings, and describing classes by only their attributes is ineffective. A viable strategy that we find to increase benefits from attributes is contrastive training with adjective-based negative captions.