SSH: A Self-Supervised Framework for Image Harmonization
It addresses the challenge of limited annotated data for image harmonization, benefiting computer vision applications like photo editing, though it is incremental as it builds on existing self-supervised and representation fusion ideas.
The paper tackles the problem of image harmonization, which matches appearance between foreground and background in composited images, by proposing a self-supervised framework (SSH) that trains on natural images without annotations, achieving state-of-the-art results in metrics, visual quality, and user studies.
Image harmonization aims to improve the quality of image compositing by matching the "appearance" (\eg, color tone, brightness and contrast) between foreground and background images. However, collecting large-scale annotated datasets for this task requires complex professional retouching. Instead, we propose a novel Self-Supervised Harmonization framework (SSH) that can be trained using just "free" natural images without being edited. We reformulate the image harmonization problem from a representation fusion perspective, which separately processes the foreground and background examples, to address the background occlusion issue. This framework design allows for a dual data augmentation method, where diverse [foreground, background, pseudo GT] triplets can be generated by cropping an image with perturbations using 3D color lookup tables (LUTs). In addition, we build a real-world harmonization dataset as carefully created by expert users, for evaluation and benchmarking purposes. Our results show that the proposed self-supervised method outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study. Code and dataset are available at \url{https://github.com/VITA-Group/SSHarmonization}.