Characterizing and Improving Stability in Neural Style Transfer
This work addresses the problem of visual flickering in video style transfer for applications in real-time video processing, representing an incremental improvement over prior methods.
The paper tackled the instability and flickering in real-time video style transfer by characterizing instability through the Gram matrix trace and introduced a recurrent convolutional network with a temporal consistency loss to overcome these issues, achieving high-quality, temporally consistent stylized videos in real-time.
Recent progress in style transfer on images has focused on improving the quality of stylized images and speed of methods. However, real-time methods are highly unstable resulting in visible flickering when applied to videos. In this work we characterize the instability of these methods by examining the solution set of the style transfer objective. We show that the trace of the Gram matrix representing style is inversely related to the stability of the method. Then, we present a recurrent convolutional network for real-time video style transfer which incorporates a temporal consistency loss and overcomes the instability of prior methods. Our networks can be applied at any resolution, do not re- quire optical flow at test time, and produce high quality, temporally consistent stylized videos in real-time.