Deep Zero-Shot Learning for Scene Sketch
This work addresses a novel problem in computer vision for scene sketch analysis, though it appears incremental as it builds on existing zero-shot learning methods with sketch-specific adaptations.
The paper tackles the problem of zero-shot learning for scene sketches by addressing the large semantic gap and need for expressive features, achieving improved performance through a deep embedding model with augmented semantic vectors and attention-based feature learning.
We introduce a novel problem of scene sketch zero-shot learning (SSZSL), which is a challenging task, since (i) different from photo, the gap between common semantic domain (e.g., word vector) and sketch is too huge to exploit common semantic knowledge as the bridge for knowledge transfer, and (ii) compared with single-object sketch, more expressive feature representation for scene sketch is required to accommodate its high-level of abstraction and complexity. To overcome these challenges, we propose a deep embedding model for scene sketch zero-shot learning. In particular, we propose the augmented semantic vector to conduct domain alignment by fusing multi-modal semantic knowledge (e.g., cartoon image, natural image, text description), and adopt attention-based network for scene sketch feature learning. Moreover, we propose a novel distance metric to improve the similarity measure during testing. Extensive experiments and ablation studies demonstrate the benefit of our sketch-specific design.