From Coarse to Fine-grained Concept based Discrimination for Phrase Detection
This work addresses phrase detection in computer vision, offering incremental improvements for tasks like image captioning or visual question answering.
The paper tackles the challenge of sampling negatives for phrase detection by introducing CFCD-Net, which uses concept-based discrimination and a fine-grained module to improve discriminative training, resulting in a 1.5-2 point mAP improvement over state-of-the-art on Flickr30K Entities and RefCOCO+ datasets.
Phrase detection requires methods to identify if a phrase is relevant to an image and localize it, if applicable. A key challenge for training more discriminative detection models is sampling negatives. Sampling techniques from prior work focus primarily on hard, often noisy, negatives disregarding the broader distribution of negative samples. Our proposed CFCD-Net addresses this through two novels methods. First, we generate groups of semantically similar words we call concepts (\eg, \{dog, cat, horse\} and \ \{car, truck, SUV\}), and then train our CFCD-Net to discriminate between a region of interest and its unrelated concepts. Second, for phrases containing fine-grained mutually-exclusive words (\eg, colors), we force the model to select only one applicable phrase for each region using our novel fine-grained module (FGM). We evaluate our approach on Flickr30K Entities and RefCOCO+, where we improve mAP over the state-of-the-art by 1.5-2 points. When considering only the phrases affected by our FGM module, we improve by 3-4 points on both datasets.