SurgGoal: Rethinking Surgical Planning Evaluation via Goal-Satisfiability
This addresses evaluation reliability for safety-critical surgical planning, though it is incremental as it refines existing methods rather than introducing a new paradigm.
The paper tackled the problem of unreliable evaluation of vision-language models in surgical planning by defining correctness via goal satisfiability and introducing a benchmark with valid and invalid plans, showing that sequence similarity metrics misjudge quality and that rule-based metrics reveal failures due to perception errors and reasoning constraints.
Surgical planning integrates visual perception, long-horizon reasoning, and procedural knowledge, yet it remains unclear whether current evaluation protocols reliably assess vision-language models (VLMs) in safety-critical settings. Motivated by a goal-oriented view of surgical planning, we define planning correctness via phase-goal satisfiability, where plan validity is determined by expert-defined surgical rules. Based on this definition, we introduce a multicentric meta-evaluation benchmark with valid procedural variations and invalid plans containing order and content errors. Using this benchmark, we show that sequence similarity metrics systematically misjudge planning quality, penalizing valid plans while failing to identify invalid ones. We therefore adopt a rule-based goal-satisfiability metric as a high-precision meta-evaluation reference to assess Video-LLMs under progressively constrained settings, revealing failures due to perception errors and under-constrained reasoning. Structural knowledge consistently improves performance, whereas semantic guidance alone is unreliable and benefits larger models only when combined with structural constraints.