Graph Neural Networks for Knowledge Enhanced Visual Representation of Paintings
This work addresses the challenge of analyzing and curating fine art by integrating visual and semantic information, offering a more efficient and effective approach for art historians and curators.
The paper tackled the problem of fine art analysis by proposing ArtSAGENet, a multimodal architecture combining Graph Neural Networks and Convolutional Neural Networks, which outperformed traditional methods in tasks like style classification and artist attribution while requiring less computational time and labeled data.
We propose ArtSAGENet, a novel multimodal architecture that integrates Graph Neural Networks (GNNs) and Convolutional Neural Networks (CNNs), to jointly learn visual and semantic-based artistic representations. First, we illustrate the significant advantages of multi-task learning for fine art analysis and argue that it is conceptually a much more appropriate setting in the fine art domain than the single-task alternatives. We further demonstrate that several GNN architectures can outperform strong CNN baselines in a range of fine art analysis tasks, such as style classification, artist attribution, creation period estimation, and tag prediction, while training them requires an order of magnitude less computational time and only a small amount of labeled data. Finally, through extensive experimentation we show that our proposed ArtSAGENet captures and encodes valuable relational dependencies between the artists and the artworks, surpassing the performance of traditional methods that rely solely on the analysis of visual content. Our findings underline a great potential of integrating visual content and semantics for fine art analysis and curation.