KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph
This work addresses the problem of automated artwork interpretation for art historians and enthusiasts, representing an incremental improvement by enhancing existing models with metadata integration.
The authors tackled the challenge of generating descriptive and interpretative captions for fine-art paintings by developing KALE, a knowledge-augmented vision-language model that integrates artwork metadata via textual input and a heterogeneous graph, achieving strong performance on multiple datasets as measured by CIDEr scores.
Exploring the narratives conveyed by fine-art paintings is a challenge in image captioning, where the goal is to generate descriptions that not only precisely represent the visual content but also offer a in-depth interpretation of the artwork's meaning. The task is particularly complex for artwork images due to their diverse interpretations and varied aesthetic principles across different artistic schools and styles. In response to this, we present KALE Knowledge-Augmented vision-Language model for artwork Elaborations), a novel approach that enhances existing vision-language models by integrating artwork metadata as additional knowledge. KALE incorporates the metadata in two ways: firstly as direct textual input, and secondly through a multimodal heterogeneous knowledge graph. To optimize the learning of graph representations, we introduce a new cross-modal alignment loss that maximizes the similarity between the image and its corresponding metadata. Experimental results demonstrate that KALE achieves strong performance (when evaluated with CIDEr, in particular) over existing state-of-the-art work across several artwork datasets. Source code of the project is available at https://github.com/Yanbei-Jiang/Artwork-Interpretation.