DesCo: Learning Object Recognition with Rich Language Descriptions
This addresses the problem of improving object recognition accuracy in AI systems for applications requiring fine-grained visual understanding, though it is incremental as it builds on existing vision-language models.
The paper tackles the problem that vision-language models often ignore contextual information in complex language descriptions and rely too heavily on object names alone, proposing a new description-conditioned paradigm (DesCo) that uses a large language model to generate rich descriptions and context-sensitive queries. On zero-shot object detection benchmarks LVIS and OminiLabel, DesCo achieves 34.8 APr minival (+9.1 improvement) and 29.3 AP (+3.6 improvement), surpassing prior state-of-the-art models GLIP and FIBER.
Recent development in vision-language approaches has instigated a paradigm shift in learning visual recognition models from language supervision. These approaches align objects with language queries (e.g. "a photo of a cat") and improve the models' adaptability to identify novel objects and domains. Recently, several studies have attempted to query these models with complex language expressions that include specifications of fine-grained semantic details, such as attributes, shapes, textures, and relations. However, simply incorporating language descriptions as queries does not guarantee accurate interpretation by the models. In fact, our experiments show that GLIP, the state-of-the-art vision-language model for object detection, often disregards contextual information in the language descriptions and instead relies heavily on detecting objects solely by their names. To tackle the challenges, we propose a new description-conditioned (DesCo) paradigm of learning object recognition models with rich language descriptions consisting of two major innovations: 1) we employ a large language model as a commonsense knowledge engine to generate rich language descriptions of objects based on object names and the raw image-text caption; 2) we design context-sensitive queries to improve the model's ability in deciphering intricate nuances embedded within descriptions and enforce the model to focus on context rather than object names alone. On two novel object detection benchmarks, LVIS and OminiLabel, under the zero-shot detection setting, our approach achieves 34.8 APr minival (+9.1) and 29.3 AP (+3.6), respectively, surpassing the prior state-of-the-art models, GLIP and FIBER, by a large margin.