Improved Style Transfer by Respecting Inter-layer Correlations
This work addresses a specific limitation in style transfer methods for handling hierarchical and long-scale patterns, representing an incremental improvement over existing approaches.
The paper tackles the problem of style transfer for images with strong structural patterns across spatial scales, such as textures with dots on curves, by controlling inter-layer correlations using cross-layer gram matrices, resulting in visible improvements in style transfer and texture synthesis.
A popular series of style transfer methods apply a style to a content image by controlling mean and covariance of values in early layers of a feature stack. This is insufficient for transferring styles that have strong structure across spatial scales like, e.g., textures where dots lie on long curves. This paper demonstrates that controlling inter-layer correlations yields visible improvements in style transfer methods. We achieve this control by computing cross-layer, rather than within-layer, gram matrices. We find that (a) cross-layer gram matrices are sufficient to control within-layer statistics. Inter-layer correlations improves style transfer and texture synthesis. The paper shows numerous examples on "hard" real style transfer problems (e.g. long scale and hierarchical patterns); (b) a fast approximate style transfer method can control cross-layer gram matrices; (c) we demonstrate that multiplicative, rather than additive style and content loss, results in very good style transfer. Multiplicative loss produces a visible emphasis on boundaries, and means that one hyper-parameter can be eliminated.