Fine-Grained Spatially Varying Material Selection in Images
This work addresses the need for precise material selection in image editing, enabling faster and simpler modifications for users, though it appears incremental as it builds on existing vision transformer approaches.
The paper tackles the problem of material selection in images by introducing a method robust to lighting and reflectance variations, using vision transformer models and multi-resolution processing to achieve finer and more stable results than prior methods, with a new dataset of over 800,000 synthetic images for texture and subtexture-level annotations.
Selection is the first step in many image editing processes, enabling faster and simpler modifications of all pixels sharing a common modality. In this work, we present a method for material selection in images, robust to lighting and reflectance variations, which can be used for downstream editing tasks. We rely on vision transformer (ViT) models and leverage their features for selection, proposing a multi-resolution processing strategy that yields finer and more stable selection results than prior methods. Furthermore, we enable selection at two levels: texture and subtexture, leveraging a new two-level material selection (DuMaS) dataset which includes dense annotations for over 800,000 synthetic images, both on the texture and subtexture levels.