StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval
This addresses a fundamental challenge in SBIR for improving retrieval accuracy across diverse human-drawn sketches, though it is incremental as it builds on existing disentanglement and meta-learning techniques.
The paper tackles the problem of style variations in sketch-based image retrieval (SBIR) by proposing a style-agnostic model that disentangles semantic content from user-specific styles using a cross-modal VAE with meta-learning, achieving state-of-the-art performance in category-level and instance-level SBIR.
Sketch-based image retrieval (SBIR) is a cross-modal matching problem which is typically solved by learning a joint embedding space where the semantic content shared between photo and sketch modalities are preserved. However, a fundamental challenge in SBIR has been largely ignored so far, that is, sketches are drawn by humans and considerable style variations exist amongst different users. An effective SBIR model needs to explicitly account for this style diversity, crucially, to generalise to unseen user styles. To this end, a novel style-agnostic SBIR model is proposed. Different from existing models, a cross-modal variational autoencoder (VAE) is employed to explicitly disentangle each sketch into a semantic content part shared with the corresponding photo, and a style part unique to the sketcher. Importantly, to make our model dynamically adaptable to any unseen user styles, we propose to meta-train our cross-modal VAE by adding two style-adaptive components: a set of feature transformation layers to its encoder and a regulariser to the disentangled semantic content latent code. With this meta-learning framework, our model can not only disentangle the cross-modal shared semantic content for SBIR, but can adapt the disentanglement to any unseen user style as well, making the SBIR model truly style-agnostic. Extensive experiments show that our style-agnostic model yields state-of-the-art performance for both category-level and instance-level SBIR.