CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets
This addresses the issue of coarsely-defined class labels in zero-shot classification for researchers and practitioners using open vocabulary models, though it is incremental as it builds on existing CLIP methods.
The paper tackles the problem of zero-shot image classification by improving class name richness for datasets with implicit semantic hierarchies, proposing CHiLS which uses hierarchical label sets to boost accuracy without additional training. Results show improved accuracy across multiple datasets, with specific gains such as a 5% increase on ImageNet-Hierarchical.
Open vocabulary models (e.g. CLIP) have shown strong performance on zero-shot classification through their ability generate embeddings for each class based on their (natural language) names. Prior work has focused on improving the accuracy of these models through prompt engineering or by incorporating a small amount of labeled downstream data (via finetuning). However, there has been little focus on improving the richness of the class names themselves, which can pose issues when class labels are coarsely-defined and are uninformative. We propose Classification with Hierarchical Label Sets (or CHiLS), an alternative strategy for zero-shot classification specifically designed for datasets with implicit semantic hierarchies. CHiLS proceeds in three steps: (i) for each class, produce a set of subclasses, using either existing label hierarchies or by querying GPT-3; (ii) perform the standard zero-shot CLIP procedure as though these subclasses were the labels of interest; (iii) map the predicted subclass back to its parent to produce the final prediction. Across numerous datasets with underlying hierarchical structure, CHiLS leads to improved accuracy in situations both with and without ground-truth hierarchical information. CHiLS is simple to implement within existing zero-shot pipelines and requires no additional training cost. Code is available at: https://github.com/acmi-lab/CHILS.