Language-guided Semantic Style Transfer of 3D Indoor Scenes
This addresses the challenge of automatically stylizing 3D indoor scenes based on textual descriptions, which could benefit multimedia applications, though it appears incremental as it builds on existing style transfer and vision-language techniques.
The paper tackles the problem of language-guided semantic style transfer for 3D indoor scenes, using a method that maps vertex coordinates to colors, renders images, compares them to phrases via vision-language models, and updates colors through back-propagation, with results validated through qualitative analyses and A/B user tests on ScanNet and SceneNN datasets, showing improved quality with semantics and HSV regularization.
We address the new problem of language-guided semantic style transfer of 3D indoor scenes. The input is a 3D indoor scene mesh and several phrases that describe the target scene. Firstly, 3D vertex coordinates are mapped to RGB residues by a multi-layer perceptron. Secondly, colored 3D meshes are differentiablly rendered into 2D images, via a viewpoint sampling strategy tailored for indoor scenes. Thirdly, rendered 2D images are compared to phrases, via pre-trained vision-language models. Lastly, errors are back-propagated to the multi-layer perceptron to update vertex colors corresponding to certain semantic categories. We did large-scale qualitative analyses and A/B user tests, with the public ScanNet and SceneNN datasets. We demonstrate: (1) visually pleasing results that are potentially useful for multimedia applications. (2) rendering 3D indoor scenes from viewpoints consistent with human priors is important. (3) incorporating semantics significantly improve style transfer quality. (4) an HSV regularization term leads to results that are more consistent with inputs and generally rated better. Codes and user study toolbox are available at https://github.com/AIR-DISCOVER/LASST