Class-Based Styling: Real-time Localized Style Transfer with Semantic Segmentation
This addresses the problem of efficient and customizable visual styling in videos for applications like video editing or augmented reality, but it is incremental as it combines existing methods.
The paper tackles real-time localized style transfer by mapping different styles to different object classes using semantic segmentation, achieving high-quality results and real-time performance at 104 FPS with 0.76 million parameters.
We propose a Class-Based Styling method (CBS) that can map different styles for different object classes in real-time. CBS achieves real-time performance by carrying out two steps simultaneously. While a semantic segmentation method is used to obtain the mask of each object class in a video frame, a styling method is used to style that frame globally. Then an object class can be styled by combining the segmentation mask and the styled image. The user can also select multiple styles so that different object classes can have different styles in a single frame. For semantic segmentation, we leverage DABNet that achieves high accuracy, yet only has 0.76 million parameters and runs at 104 FPS. For the style transfer step, we use a popular real-time method proposed by Johnson et al. [7]. We evaluated CBS on a video of the CityScapes dataset and observed high-quality localized style transfer results for different object classes and real-time performance.