CAD-Prompted SAM3: Geometry-Conditioned Instance Segmentation for Industrial Objects
This addresses segmentation challenges in manufacturing and 3D printing by leveraging precise geometric data, though it is incremental as it builds on SAM3 with a novel prompting method.
The paper tackles the problem of segmenting industrial objects where natural language or appearance-based prompts are unreliable, by proposing a CAD-prompted segmentation framework using multi-view renderings of CAD models as prompts, achieving single-stage mask prediction for objects with varying materials and finishes.
Verbal-prompted segmentation is inherently limited by the expressiveness of natural language and struggles with uncommon, instance-specific, or difficult-to-describe objects: scenarios frequently encountered in manufacturing and 3D printing environments. While image exemplars provide an alternative, they primarily encode appearance cues such as color and texture, which are often unrelated to a part's geometric identity. In industrial settings, a single component may be produced in different materials, finishes, or colors, making appearance-based prompting unreliable. In contrast, such objects are typically defined by precise CAD models that capture their canonical geometry. We propose a CAD-prompted segmentation framework built on SAM3 that uses canonical multi-view renderings of a CAD model as prompt input. The rendered views provide geometry-based conditioning independent of surface appearance. The model is trained using synthetic data generated from mesh renderings in simulation under diverse viewpoints and scene contexts. Our approach enables single-stage, CAD-prompted mask prediction, extending promptable segmentation to objects that cannot be robustly described by language or appearance alone.