Surgical Triplet Recognition via Diffusion Model
This addresses the problem of context-aware operating rooms for surgeons, but it is incremental as it applies an existing diffusion model to a specific domain.
The paper tackles surgical triplet recognition in videos by proposing DiffTriplet, a generative framework using a diffusion model with association learning and guidance, achieving new state-of-the-art performance on CholecT45 and CholecT50 datasets.
Surgical triplet recognition is an essential building block to enable next-generation context-aware operating rooms. The goal is to identify the combinations of instruments, verbs, and targets presented in surgical video frames. In this paper, we propose DiffTriplet, a new generative framework for surgical triplet recognition employing the diffusion model, which predicts surgical triplets via iterative denoising. To handle the challenge of triplet association, two unique designs are proposed in our diffusion framework, i.e., association learning and association guidance. During training, we optimize the model in the joint space of triplets and individual components to capture the dependencies among them. At inference, we integrate association constraints into each update of the iterative denoising process, which refines the triplet prediction using the information of individual components. Experiments on the CholecT45 and CholecT50 datasets show the superiority of the proposed method in achieving a new state-of-the-art performance for surgical triplet recognition. Our codes will be released.