Son of Zorn's Lemma: Targeted Style Transfer Using Instance-aware Semantic Segmentation
This addresses a problem for artists and augmented reality applications, such as in films or games, by enabling precise object-level style changes, though it appears incremental as it builds on existing style transfer techniques.
The paper tackles targeted style transfer, where only a specific object in an image is stylized without altering its surroundings, by presenting a method that simultaneously segments and stylizes objects using a Markov random field model for smooth blending.
Style transfer is an important task in which the style of a source image is mapped onto that of a target image. The method is useful for synthesizing derivative works of a particular artist or specific painting. This work considers targeted style transfer, in which the style of a template image is used to alter only part of a target image. For example, an artist may wish to alter the style of only one particular object in a target image without altering the object's general morphology or surroundings. This is useful, for example, in augmented reality applications (such as the recently released Pokemon GO), where one wants to alter the appearance of a single real-world object in an image frame to make it appear as a cartoon. Most notably, the rendering of real-world objects into cartoon characters has been used in a number of films and television show, such as the upcoming series Son of Zorn. We present a method for targeted style transfer that simultaneously segments and stylizes single objects selected by the user. The method uses a Markov random field model to smooth and anti-alias outlier pixels near object boundaries, so that stylized objects naturally blend into their surroundings.