Let's Transfer Transformations of Shared Semantic Representations
This enables image manipulation and retrieval with desired modifications for applications like zero-shot learning and attribute composition, though it is incremental as it builds on shared semantic representation concepts.
The paper tackles the problem of learning image transformation operations without training examples by transferring transformations learned in another domain, demonstrating this approach on image retrieval tasks where queries combine an image with a transformation specification. Results show successful transfer from synthesized 2D blobs to 3D rendered images and from text to natural images.
With a good image understanding capability, can we manipulate the images high level semantic representation? Such transformation operation can be used to generate or retrieve similar images but with a desired modification (for example changing beach background to street background); similar ability has been demonstrated in zero shot learning, attribute composition and attribute manipulation image search. In this work we show how one can learn transformations with no training examples by learning them on another domain and then transfer to the target domain. This is feasible if: first, transformation training data is more accessible in the other domain and second, both domains share similar semantics such that one can learn transformations in a shared embedding space. We demonstrate this on an image retrieval task where search query is an image, plus an additional transformation specification (for example: search for images similar to this one but background is a street instead of a beach). In one experiment, we transfer transformation from synthesized 2D blobs image to 3D rendered image, and in the other, we transfer from text domain to natural image domain.