Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features
This provides a class-agnostic, training-free method for decorating untextured shapes with semantic features, addressing a bottleneck in 3D shape analysis, though it is incremental as it builds on existing diffusion and image foundational models.
The authors tackled the problem of generating semantic feature descriptors for untextured 3D shapes by distilling diffusion features from image models, achieving robust correspondence across isometric and non-isometric shape families on benchmarks like SHREC'19 and FAUST.
We present Diff3F as a simple, robust, and class-agnostic feature descriptor that can be computed for untextured input shapes (meshes or point clouds). Our method distills diffusion features from image foundational models onto input shapes. Specifically, we use the input shapes to produce depth and normal maps as guidance for conditional image synthesis. In the process, we produce (diffusion) features in 2D that we subsequently lift and aggregate on the original surface. Our key observation is that even if the conditional image generations obtained from multi-view rendering of the input shapes are inconsistent, the associated image features are robust and, hence, can be directly aggregated across views. This produces semantic features on the input shapes, without requiring additional data or training. We perform extensive experiments on multiple benchmarks (SHREC'19, SHREC'20, FAUST, and TOSCA) and demonstrate that our features, being semantic instead of geometric, produce reliable correspondence across both isometric and non-isometrically related shape families. Code is available via the project page at https://diff3f.github.io/