SAMa: Material-aware 3D Selection and Segmentation
This addresses the problem of manual material segmentation for artists and creators, offering an incremental improvement by adapting an existing video model to the 3D material domain.
The paper tackles the manual decomposition of 3D assets into material parts by introducing SAMa, a material selection approach that extends SAM2 to create 3D-consistent material-similarity representations, enabling optimization-free selection in seconds and outperforming baselines in accuracy and consistency.
Decomposing 3D assets into material parts is a common task for artists and creators, yet remains a highly manual process. In this work, we introduce Select Any Material (SAMa), a material selection approach for various 3D representations. Building on the recently introduced SAM2 video selection model, we extend its capabilities to the material domain. We leverage the model's cross-view consistency to create a 3D-consistent intermediate material-similarity representation in the form of a point cloud from a sparse set of views. Nearest-neighbour lookups in this similarity cloud allow us to efficiently reconstruct accurate continuous selection masks over objects' surfaces that can be inspected from any view. Our method is multiview-consistent by design, alleviating the need for contrastive learning or feature-field pre-processing, and performs optimization-free selection in seconds. Our approach works on arbitrary 3D representations and outperforms several strong baselines in terms of selection accuracy and multiview consistency. It enables several compelling applications, such as replacing the diffuse-textured materials on a text-to-3D output, or selecting and editing materials on NeRFs and 3D-Gaussians.