Dynamic Multi-level Weighted Alignment Network for Zero-shot Sketch-based Image Retrieval
This work improves retrieval accuracy for applications such as e-commerce, but it is incremental as it builds on existing ZS-SBIR methods with specific enhancements.
The paper tackles the problem of zero-shot sketch-based image retrieval by addressing imbalanced modality samples and inconsistent low-quality information, resulting in superior performance over state-of-the-art methods on benchmark datasets like Sketchy, TU-Berlin, and QuickDraw.
The problem of zero-shot sketch-based image retrieval (ZS-SBIR) has achieved increasing attention due to its wide applications, e.g. e-commerce. Despite progress made in this field, previous works suffer from using imbalanced samples of modalities and inconsistent low-quality information during training, resulting in sub-optimal performance. Therefore, in this paper, we introduce an approach called Dynamic Multi-level Weighted Alignment Network for ZS-SBIR. It consists of three components: (i) a Uni-modal Feature Extraction Module that includes a CLIP text encoder and a ViT for extracting textual and visual tokens, (ii) a Cross-modal Multi-level Weighting Module that produces an alignment weight list by the local and global aggregation blocks to measure the aligning quality of sketch and image samples, (iii) a Weighted Quadruplet Loss Module aiming to improve the balance of domains in the triplet loss. Experiments on three benchmark datasets, i.e., Sketchy, TU-Berlin, and QuickDraw, show our method delivers superior performances over the state-of-the-art ZS-SBIR methods.