Towards Recognizing Unseen Categories in Unseen Domains
This addresses a critical challenge in deep visual recognition for applications requiring robustness to new classes and scenarios, though it builds incrementally on existing ZSL and DG techniques.
The paper tackles the joint problem of recognizing unseen categories in unseen domains by proposing CuMix, a curriculum-based mixup method that simulates test-time shifts, achieving state-of-the-art results on benchmarks like DomainNet.
Current deep visual recognition systems suffer from severe performance degradation when they encounter new images from classes and scenarios unseen during training. Hence, the core challenge of Zero-Shot Learning (ZSL) is to cope with the semantic-shift whereas the main challenge of Domain Adaptation and Domain Generalization (DG) is the domain-shift. While historically ZSL and DG tasks are tackled in isolation, this work develops with the ambitious goal of solving them jointly,i.e. by recognizing unseen visual concepts in unseen domains. We presentCuMix (CurriculumMixup for recognizing unseen categories in unseen domains), a holistic algorithm to tackle ZSL, DG and ZSL+DG. The key idea of CuMix is to simulate the test-time domain and semantic shift using images and features from unseen domains and categories generated by mixing up the multiple source domains and categories available during training. Moreover, a curriculum-based mixing policy is devised to generate increasingly complex training samples. Results on standard SL and DG datasets and on ZSL+DG using the DomainNet benchmark demonstrate the effectiveness of our approach.