ArtContext: Contextualizing Artworks with Open-Access Art History Articles and Wikidata Knowledge through a LoRA-Tuned CLIP Model
This addresses the challenge for art historians and viewers in accessing contextual information about artworks, though it is incremental as it adapts existing methods to a new domain.
The researchers tackled the problem of identifying relevant art history information for artworks by developing ArtContext, a pipeline that uses open-access articles and Wikidata to annotate artworks, resulting in PaintingCLIP, a domain-specific model that outperforms CLIP.
Many Art History articles discuss artworks in general as well as specific parts of works, such as layout, iconography, or material culture. However, when viewing an artwork, it is not trivial to identify what different articles have said about the piece. Therefore, we propose ArtContext, a pipeline for taking a corpus of Open-Access Art History articles and Wikidata Knowledge and annotating Artworks with this information. We do this using a novel corpus collection pipeline, then learn a bespoke CLIP model adapted using Low-Rank Adaptation (LoRA) to make it domain-specific. We show that the new model, PaintingCLIP, which is weakly supervised by the collected corpus, outperforms CLIP and provides context for a given artwork. The proposed pipeline is generalisable and can be readily applied to numerous humanities areas.