NLUT: Neural-based 3D Lookup Tables for Video Photorealistic Style Transfer
This work addresses the need for fast and consistent video style transfer, which is incremental as it builds on existing photorealistic style transfer methods by enhancing speed and temporal coherence.
The paper tackles the problem of video photorealistic style transfer by proposing neural-based 3D lookup tables to improve efficiency and temporal consistency, achieving processing speeds of less than 2 milliseconds per frame for 8K video and outperforming existing methods in visual quality and consistency.
Video photorealistic style transfer is desired to generate videos with a similar photorealistic style to the style image while maintaining temporal consistency. However, existing methods obtain stylized video sequences by performing frame-by-frame photorealistic style transfer, which is inefficient and does not ensure the temporal consistency of the stylized video. To address this issue, we use neural network-based 3D Lookup Tables (LUTs) for the photorealistic transfer of videos, achieving a balance between efficiency and effectiveness. We first train a neural network for generating photorealistic stylized 3D LUTs on a large-scale dataset; then, when performing photorealistic style transfer for a specific video, we select a keyframe and style image in the video as the data source and fine-turn the neural network; finally, we query the 3D LUTs generated by the fine-tuned neural network for the colors in the video, resulting in a super-fast photorealistic style transfer, even processing 8K video takes less than 2 millisecond per frame. The experimental results show that our method not only realizes the photorealistic style transfer of arbitrary style images but also outperforms the existing methods in terms of visual quality and consistency. Project page:https://semchan.github.io/NLUT_Project.