Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Models
This work addresses the need for specialized AI in surgical contexts, representing an incremental advancement by applying existing LVLM methods to a new domain.
The authors tackled the problem of understanding surgical scenarios by developing a large vision-language model (LVLM) specifically designed for surgical images and videos, which demonstrated superior performance on surgical visual question-answering datasets compared to previous works.
Conversation agents powered by large language models are revolutionizing the way we interact with visual data. Recently, large vision-language models (LVLMs) have been extensively studied for both images and videos. However, these studies typically focus on common scenarios. In this work, we introduce an LVLM specifically designed for surgical scenarios. We integrate visual representations of surgical images and videos into the language feature space. Consequently, we establish a LVLM model, Surgical-LLaVA, fine-tuned on instruction following data of surgical scenarios. Our experiments demonstrate that Surgical-LLaVA exhibits impressive multi-modal chat abilities in surgical contexts, occasionally displaying multi-modal behaviors on unseen instructions. We conduct a quantitative evaluation of visual question-answering datasets for surgical scenarios. The results show superior performance compared to previous works, indicating the potential of our model to tackle more complex surgery scenarios.