StyleRF-VolVis: Style Transfer of Neural Radiance Fields for Expressive Volume Visualization
This addresses the need for efficient and high-quality expressive volume visualization in fields like medical imaging or scientific data analysis, though it appears incremental as it builds on existing NeRF-based approaches.
The paper tackles the problem of generating novel visualizations in volume visualization by introducing StyleRF-VolVis, a style transfer framework that uses neural radiance fields to separate scene geometry and color appearance, enabling flexible editing while maintaining consistency, and demonstrates superior quality compared to existing methods.
In volume visualization, visualization synthesis has attracted much attention due to its ability to generate novel visualizations without following the conventional rendering pipeline. However, existing solutions based on generative adversarial networks often require many training images and take significant training time. Still, issues such as low quality, consistency, and flexibility persist. This paper introduces StyleRF-VolVis, an innovative style transfer framework for expressive volume visualization (VolVis) via neural radiance field (NeRF). The expressiveness of StyleRF-VolVis is upheld by its ability to accurately separate the underlying scene geometry (i.e., content) and color appearance (i.e., style), conveniently modify color, opacity, and lighting of the original rendering while maintaining visual content consistency across the views, and effectively transfer arbitrary styles from reference images to the reconstructed 3D scene. To achieve these, we design a base NeRF model for scene geometry extraction, a palette color network to classify regions of the radiance field for photorealistic editing, and an unrestricted color network to lift the color palette constraint via knowledge distillation for non-photorealistic editing. We demonstrate the superior quality, consistency, and flexibility of StyleRF-VolVis by experimenting with various volume rendering scenes and reference images and comparing StyleRF-VolVis against other image-based (AdaIN), video-based (ReReVST), and NeRF-based (ARF and SNeRF) style rendering solutions.