CoTDet: Affordance Knowledge Prompting for Task Driven Object Detection
This work addresses the problem of detecting objects for specific tasks in computer vision, offering a novel approach that improves accuracy and provides rationales, though it is incremental in advancing detection methods.
The paper tackles the challenge of task-driven object detection, where object categories are too diverse for traditional methods, by proposing CoTDet, a framework that uses affordance knowledge prompting from large language models to condition detection, resulting in significant performance gains of +15.6 box AP and +14.8 mask AP over state-of-the-art methods.
Task driven object detection aims to detect object instances suitable for affording a task in an image. Its challenge lies in object categories available for the task being too diverse to be limited to a closed set of object vocabulary for traditional object detection. Simply mapping categories and visual features of common objects to the task cannot address the challenge. In this paper, we propose to explore fundamental affordances rather than object categories, i.e., common attributes that enable different objects to accomplish the same task. Moreover, we propose a novel multi-level chain-of-thought prompting (MLCoT) to extract the affordance knowledge from large language models, which contains multi-level reasoning steps from task to object examples to essential visual attributes with rationales. Furthermore, to fully exploit knowledge to benefit object recognition and localization, we propose a knowledge-conditional detection framework, namely CoTDet. It conditions the detector from the knowledge to generate object queries and regress boxes. Experimental results demonstrate that our CoTDet outperforms state-of-the-art methods consistently and significantly (+15.6 box AP and +14.8 mask AP) and can generate rationales for why objects are detected to afford the task.