Enhancing Novel Object Detection via Cooperative Foundational Models
This addresses the challenge of detecting both known and novel objects for computer vision applications, representing a strong specific gain rather than a foundational breakthrough.
The paper tackles the problem of novel object detection by transforming closed-set detectors into open-set ones using a cooperative mechanism with foundational models like CLIP and SAM, achieving 17.42 mAP for novel objects and 42.08 mAP for known objects on LVIS, and surpassing state-of-the-art by 7.2 AP50 on COCO OVD.
In this work, we address the challenging and emergent problem of novel object detection (NOD), focusing on the accurate detection of both known and novel object categories during inference. Traditional object detection algorithms are inherently closed-set, limiting their capability to handle NOD. We present a novel approach to transform existing closed-set detectors into open-set detectors. This transformation is achieved by leveraging the complementary strengths of pre-trained foundational models, specifically CLIP and SAM, through our cooperative mechanism. Furthermore, by integrating this mechanism with state-of-the-art open-set detectors such as GDINO, we establish new benchmarks in object detection performance. Our method achieves 17.42 mAP in novel object detection and 42.08 mAP for known objects on the challenging LVIS dataset. Adapting our approach to the COCO OVD split, we surpass the current state-of-the-art by a margin of 7.2 $ \text{AP}_{50} $ for novel classes. Our code is available at https://rohit901.github.io/coop-foundation-models/ .