3-D Material Style Transfer for Reconstructing Unknown Appearance in Complex Natural Materials
This work addresses the challenge of reconstructing unknown appearance in materials like eggshells and fossils, with applications in paleontology and artifact restoration, but it is incremental as it builds on existing style transfer and material analysis techniques.
The paper tackles the problem of reconstructing invisible or faded appearance in complex natural materials by proposing a 3-D material style transfer framework that transfers appearance properties from an exemplar to a target object with noncorresponding color patterns, achieving more accurate color pattern replication than methods relying on shape correspondences and coarse perceptual differences.
We propose a 3-D material style transfer framework for reconstructing invisible (or faded) appearance properties in complex natural materials. Our algorithm addresses the technical challenge of transferring appearance properties from one object to another of the same material when both objects have intricate, noncorresponding color patterns. Eggshells, exoskeletons, and minerals, for example, have patterns composed of highly randomized layers of organic and inorganic compounds. These materials pose a challenge as the distribution of compounds that determine surface color changes from object to object and within local pattern regions. Our solution adapts appearance observations from a material property distribution in an exemplar to the material property distribution of a target object to reconstruct its unknown appearance. We use measured reflectance in 3-D bispectral textures to record changing material property distributions. Our novel implementation of spherical harmonics uses principles from chemistry and biology to learn relationships between color (hue and saturation) and material composition and concentration in an exemplar. The encoded relationships are transformed to the property distribution of a target for color recovery and material assignment. Quantitative and qualitative evaluation methods show that we replicate color patterns more accurately than methods that only rely on shape correspondences and coarse-level perceptual differences. We demonstrate applications of our work for reconstructing color in extinct fossils, restoring faded artifacts and generating synthetic textures.