Insights From A Large-Scale Database of Material Depictions In Paintings
This work provides insights into the cross-domain applicability of visual recognition systems between natural images and fine art, which is significant for researchers working on computer vision and art analysis.
This paper explores the application of deep learning recognition systems, typically used for natural images, to paintings and vice-versa. It demonstrates that existing tools like interactive segmentation and FasterRCNN can be effectively repurposed for tasks like material detection in paintings, and that training on paintings can improve feature quality for networks intended for natural images.
Deep learning has paved the way for strong recognition systems which are often both trained on and applied to natural images. In this paper, we examine the give-and-take relationship between such visual recognition systems and the rich information available in the fine arts. First, we find that visual recognition systems designed for natural images can work surprisingly well on paintings. In particular, we find that interactive segmentation tools can be used to cleanly annotate polygonal segments within paintings, a task which is time consuming to undertake by hand. We also find that FasterRCNN, a model which has been designed for object recognition in natural scenes, can be quickly repurposed for detection of materials in paintings. Second, we show that learning from paintings can be beneficial for neural networks that are intended to be used on natural images. We find that training on paintings instead of natural images can improve the quality of learned features and we further find that a large number of paintings can be a valuable source of test data for evaluating domain adaptation algorithms. Our experiments are based on a novel large-scale annotated database of material depictions in paintings which we detail in a separate manuscript.