Towards virtual painting recolouring using Vision Transformer on X-Ray Fluorescence datacubes
This work tackles the problem of digitally recoloring paintings for art conservation or analysis, but it appears incremental as it builds on existing methods with specific adaptations for XRF data.
The researchers developed a pipeline for virtual painting recoloring using X-Ray Fluorescence (XRF) data on artworks, addressing small dataset size by generating synthetic data and using a Deep Variational Embedding network to reduce dimensionality. They reported performance in terms of visual quality metrics.
In this contribution, we define (and test) a pipeline to perform virtual painting recolouring using raw data of X-Ray Fluorescence (XRF) analysis on pictorial artworks. To circumvent the small dataset size, we generate a synthetic dataset, starting from a database of XRF spectra; furthermore, to ensure a better generalisation capacity (and to tackle the issue of in-memory size and inference time), we define a Deep Variational Embedding network to embed the XRF spectra into a lower dimensional, K-Means friendly, metric space. We thus train a set of models to assign coloured images to embedded XRF images. We report here the devised pipeline performances in terms of visual quality metrics, and we close on a discussion on the results.