Exploiting CLIP-based Multi-modal Approach for Artwork Classification and Retrieval
This work addresses artwork analysis tasks for cultural heritage and digital art communities, but it is incremental as it applies an existing method to a new domain.
The authors tackled artwork classification and retrieval by applying the CLIP model to the NoisyArt dataset, achieving impressive zero-shot classification results and promising retrieval performance.
Given the recent advances in multimodal image pretraining where visual models trained with semantically dense textual supervision tend to have better generalization capabilities than those trained using categorical attributes or through unsupervised techniques, in this work we investigate how recent CLIP model can be applied in several tasks in artwork domain. We perform exhaustive experiments on the NoisyArt dataset which is a dataset of artwork images crawled from public resources on the web. On such dataset CLIP achieves impressive results on (zero-shot) classification and promising results in both artwork-to-artwork and description-to-artwork domain.