LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors
This work improves open-vocabulary object detection for computer vision applications by leveraging fine-grained descriptors, representing an incremental advancement over existing distillation-based approaches.
The paper tackles the problem of open-vocabulary object detection by addressing the limitation of existing methods that align region embeddings only with categorical labels, ignoring fine-grained object part descriptions. The result is DVDet, which outperforms state-of-the-art methods by large margins on multiple large-scale benchmarks.
Inspired by the outstanding zero-shot capability of vision language models (VLMs) in image classification tasks, open-vocabulary object detection has attracted increasing interest by distilling the broad VLM knowledge into detector training. However, most existing open-vocabulary detectors learn by aligning region embeddings with categorical labels (e.g., bicycle) only, disregarding the capability of VLMs on aligning visual embeddings with fine-grained text description of object parts (e.g., pedals and bells). This paper presents DVDet, a Descriptor-Enhanced Open Vocabulary Detector that introduces conditional context prompts and hierarchical textual descriptors that enable precise region-text alignment as well as open-vocabulary detection training in general. Specifically, the conditional context prompt transforms regional embeddings into image-like representations that can be directly integrated into general open vocabulary detection training. In addition, we introduce large language models as an interactive and implicit knowledge repository which enables iterative mining and refining visually oriented textual descriptors for precise region-text alignment. Extensive experiments over multiple large-scale benchmarks show that DVDet outperforms the state-of-the-art consistently by large margins.