Generative Model for Zero-Shot Sketch-Based Image Retrieval
This addresses the limitation of existing methods that rely on class-wise correspondences and perform poorly on novel classes, offering a solution for zero-shot sketch-based image retrieval.
The paper tackles the problem of sketch-based image retrieval for novel classes not seen during training by proposing a generative model that learns to generate images conditioned on a sketch, reducing it to an image-to-image search. The approach significantly outperforms baselines on challenging datasets like Sketchy and TU Berlin with novel train-test splits.
We present a probabilistic model for Sketch-Based Image Retrieval (SBIR) where, at retrieval time, we are given sketches from novel classes, that were not present at training time. Existing SBIR methods, most of which rely on learning class-wise correspondences between sketches and images, typically work well only for previously seen sketch classes, and result in poor retrieval performance on novel classes. To address this, we propose a generative model that learns to generate images, conditioned on a given novel class sketch. This enables us to reduce the SBIR problem to a standard image-to-image search problem. Our model is based on an inverse auto-regressive flow based variational autoencoder, with a feedback mechanism to ensure robust image generation. We evaluate our model on two very challenging datasets, Sketchy, and TU Berlin, with novel train-test split. The proposed approach significantly outperforms various baselines on both the datasets.