SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understanding
This provides a standardized benchmark for developing interactive multimodal LLMs in surgical AI, addressing inconsistent evaluation in the field.
The authors tackled the lack of unified multimodal benchmarks for surgical scene understanding by creating SurgMLLMBench, which integrates pixel-level segmentation and structured VQA annotations across multiple surgical domains, and showed that a single model trained on it achieves consistent performance and generalizes to unseen datasets.
Recent advances in multimodal large language models (LLMs) have highlighted their potential for medical and surgical applications. However, existing surgical datasets predominantly adopt a Visual Question Answering (VQA) format with heterogeneous taxonomies and lack support for pixel-level segmentation, limiting consistent evaluation and applicability. We present SurgMLLMBench, a unified multimodal benchmark explicitly designed for developing and evaluating interactive multimodal LLMs for surgical scene understanding, including the newly collected Micro-surgical Artificial Vascular anastomosIS (MAVIS) dataset. It integrates pixel-level instrument segmentation masks and structured VQA annotations across laparoscopic, robot-assisted, and micro-surgical domains under a unified taxonomy, enabling comprehensive evaluation beyond traditional VQA tasks and richer visual-conversational interactions. Extensive baseline experiments show that a single model trained on SurgMLLMBench achieves consistent performance across domains and generalizes effectively to unseen datasets. SurgMLLMBench will be publicly released as a robust resource to advance multimodal surgical AI research, supporting reproducible evaluation and development of interactive surgical reasoning models.