Refining 3D Medical Segmentation with Verbal Instruction
This addresses the challenge of suboptimal automated 3D segmentation in clinical settings, enabling clinician-in-the-loop refinement through language, though it is incremental as it builds on existing shape representation and interaction methods.
The paper tackles the problem of inaccurate 3D medical segmentation by introducing a benchmark with synthesized errors and corresponding verbal instructions, and proposes an iterative refinement model that uses these instructions to update shapes, achieving significant improvements over corrupted inputs and competitive baselines.
Accurate 3D anatomical segmentation is essential for clinical diagnosis and surgical planning. However, automated models frequently generate suboptimal shape predictions due to factors such as limited and imbalanced training data, inadequate labeling quality, and distribution shifts between training and deployment settings. A natural solution is to iteratively refine the predicted shape based on the radiologists' verbal instructions. However, this is hindered by the scarcity of paired data that explicitly links erroneous shapes to corresponding corrective instructions. As an initial step toward addressing this limitation, we introduce CoWTalk, a benchmark comprising 3D arterial anatomies with controllable synthesized anatomical errors and their corresponding repairing instructions. Building on this benchmark, we further propose an iterative refinement model that represents 3D shapes as vector sets and interacts with textual instructions to progressively update the target shape. Experimental results demonstrate that our method achieves significant improvements over corrupted inputs and competitive baselines, highlighting the feasibility of language-driven clinician-in-the-loop refinement for 3D medical shapes modeling.