Kunster -- AR Art Video Maker -- Real time video neural style transfer on mobile devices
This work makes neural style transfer more accessible for non-expert users on mobile devices, though it is incremental in applying existing techniques to mobile platforms.
The authors tackled the problems of usability and hardware requirements in neural style transfer by developing a real-time video system that runs on mobile devices at over 25 FPS, with performance tested on iPhone 11 Pro and iPhone 6s.
Neural style transfer is a well-known branch of deep learning research, with many interesting works and two major drawbacks. Most of the works in the field are hard to use by non-expert users and substantial hardware resources are required. In this work, we present a solution to both of these problems. We have applied neural style transfer to real-time video (over 25 frames per second), which is capable of running on mobile devices. We also investigate the works on achieving temporal coherence and present the idea of fine-tuning, already trained models, to achieve stable video. What is more, we also analyze the impact of the common deep neural network architecture on the performance of mobile devices with regard to number of layers and filters present. In the experiment section we present the results of our work with respect to the iOS devices and discuss the problems present in current Android devices as well as future possibilities. At the end we present the qualitative results of stylization and quantitative results of performance tested on the iPhone 11 Pro and iPhone 6s. The presented work is incorporated in Kunster - AR Art Video Maker application available in the Apple's App Store.