Style-Aware Normalized Loss for Improving Arbitrary Style Transfer
This addresses a key usability issue in neural style transfer for applications like image editing and creative tools, though it is an incremental improvement on existing AST methods.
The paper tackled the problem of imbalanced style transferability in Arbitrary Style Transfer (AST), where over 50% of stylized images were unacceptable due to under- or over-stylization, and proposed a new loss function that achieved over 80% relative improvement in style deception rate and 98% higher preference in human evaluation.
Neural Style Transfer (NST) has quickly evolved from single-style to infinite-style models, also known as Arbitrary Style Transfer (AST). Although appealing results have been widely reported in literature, our empirical studies on four well-known AST approaches (GoogleMagenta, AdaIN, LinearTransfer, and SANet) show that more than 50% of the time, AST stylized images are not acceptable to human users, typically due to under- or over-stylization. We systematically study the cause of this imbalanced style transferability (IST) and propose a simple yet effective solution to mitigate this issue. Our studies show that the IST issue is related to the conventional AST style loss, and reveal that the root cause is the equal weightage of training samples irrespective of the properties of their corresponding style images, which biases the model towards certain styles. Through investigation of the theoretical bounds of the AST style loss, we propose a new loss that largely overcomes IST. Theoretical analysis and experimental results validate the effectiveness of our loss, with over 80% relative improvement in style deception rate and 98% relatively higher preference in human evaluation.