Memory-guided Unsupervised Image-to-image Translation
This work addresses the problem of accurate style transfer for images with multiple objects in unsupervised image-to-image translation, which is incremental as it builds on existing methods by introducing a memory-based approach to handle local style variations.
The paper tackles the problem of instance-level image-to-image translation by addressing failures in handling images with multiple disparate objects due to global style application, proposing a class-aware memory network that records and accesses class-wise style variations without an object detector at test time. The model outperforms recent instance-level methods and achieves state-of-the-art performance, as demonstrated in experimental results.
We present a novel unsupervised framework for instance-level image-to-image translation. Although recent advances have been made by incorporating additional object annotations, existing methods often fail to handle images with multiple disparate objects. The main cause is that, during inference, they apply a global style to the whole image and do not consider the large style discrepancy between instance and background, or within instances. To address this problem, we propose a class-aware memory network that explicitly reasons about local style variations. A key-values memory structure, with a set of read/update operations, is introduced to record class-wise style variations and access them without requiring an object detector at the test time. The key stores a domain-agnostic content representation for allocating memory items, while the values encode domain-specific style representations. We also present a feature contrastive loss to boost the discriminative power of memory items. We show that by incorporating our memory, we can transfer class-aware and accurate style representations across domains. Experimental results demonstrate that our model outperforms recent instance-level methods and achieves state-of-the-art performance.