Cross-modal Hallucination for Few-shot Fine-grained Recognition
This addresses the challenge of limited data in fine-grained classification for computer vision applications, but it is incremental as it builds on existing multimodal and GAN-based methods.
The paper tackles the problem of few-shot fine-grained recognition by proposing a multimodal approach that uses cross-modal data hallucination to generate samples for novel classes with few examples, improving accuracy on the CUB dataset for 1-, 2-, and 5-shot learning.
State-of-the-art deep learning algorithms generally require large amounts of data for model training. Lack thereof can severely deteriorate the performance, particularly in scenarios with fine-grained boundaries between categories. To this end, we propose a multimodal approach that facilitates bridging the information gap by means of meaningful joint embeddings. Specifically, we present a benchmark that is multimodal during training (i.e. images and texts) and single-modal in testing time (i.e. images), with the associated task to utilize multimodal data in base classes (with many samples), to learn explicit visual classifiers for novel classes (with few samples). Next, we propose a framework built upon the idea of cross-modal data hallucination. In this regard, we introduce a discriminative text-conditional GAN for sample generation with a simple self-paced strategy for sample selection. We show the results of our proposed discriminative hallucinated method for 1-, 2-, and 5- shot learning on the CUB dataset, where the accuracy is improved by employing multimodal data.