Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data
This addresses the challenge of reducing manual annotation needs for surgical AI, offering a scalable solution for medical professionals, though it is incremental in applying weakly-supervised methods to a specific domain.
The paper tackles the problem of recognizing surgical objects across diverse procedures by introducing RASO, a foundation model that uses weakly-supervised learning from unannotated videos, achieving improvements of up to 10.6 mAP in zero-shot settings on surgical benchmarks.
We present RASO, a foundation model designed to Recognize Any Surgical Object, offering robust open-set recognition capabilities across a broad range of surgical procedures and object classes, in both surgical images and videos. RASO leverages a novel weakly-supervised learning framework that generates tag-image-text pairs automatically from large-scale unannotated surgical lecture videos, significantly reducing the need for manual annotations. Our scalable data generation pipeline gathers 2,200 surgical procedures and produces 3.6 million tag annotations across 2,066 unique surgical tags. Our experiments show that RASO achieves improvements of 2.9 mAP, 4.5 mAP, 10.6 mAP, and 7.2 mAP on four standard surgical benchmarks, respectively, in zero-shot settings, and surpasses state-of-the-art models in supervised surgical action recognition tasks. Code, model, and demo are available at https://ntlm1686.github.io/raso.