A New Dimension in Testimony: Relighting Video with Reflectance Field Exemplars
This enables more immersive video testimonies for historical preservation, though it is an incremental improvement over existing relighting methods.
The paper tackles the problem of estimating a person's 4D reflectance field from video footage to enable realistic relighting, achieving state-of-the-art results in realism and speed on Holocaust survivor testimonies.
We present a learning-based method for estimating 4D reflectance field of a person given video footage illuminated under a flat-lit environment of the same subject. For training data, we use one light at a time to illuminate the subject and capture the reflectance field data in a variety of poses and viewpoints. We estimate the lighting environment of the input video footage and use the subject's reflectance field to create synthetic images of the subject illuminated by the input lighting environment. We then train a deep convolutional neural network to regress the reflectance field from the synthetic images. We also use a differentiable renderer to provide feedback for the network by matching the relit images with the input video frames. This semi-supervised training scheme allows the neural network to handle unseen poses in the dataset as well as compensate for the lighting estimation error. We evaluate our method on the video footage of the real Holocaust survivors and show that our method outperforms the state-of-the-art methods in both realism and speed.