RelightableHands: Efficient Neural Relighting of Articulated Hand Models
This work addresses the challenge of efficient and realistic hand rendering for applications like VR/AR, though it is incremental as it builds on existing neural relighting and teacher-student methods.
The paper tackles the problem of rendering high-fidelity, personalized hands in real-time under novel illumination by introducing a neural relighting approach that uses a teacher-student framework with physics-inspired features, achieving photorealistic results for two interacting hands at real-time speeds.
We present the first neural relighting approach for rendering high-fidelity personalized hands that can be animated in real-time under novel illumination. Our approach adopts a teacher-student framework, where the teacher learns appearance under a single point light from images captured in a light-stage, allowing us to synthesize hands in arbitrary illuminations but with heavy compute. Using images rendered by the teacher model as training data, an efficient student model directly predicts appearance under natural illuminations in real-time. To achieve generalization, we condition the student model with physics-inspired illumination features such as visibility, diffuse shading, and specular reflections computed on a coarse proxy geometry, maintaining a small computational overhead. Our key insight is that these features have strong correlation with subsequent global light transport effects, which proves sufficient as conditioning data for the neural relighting network. Moreover, in contrast to bottleneck illumination conditioning, these features are spatially aligned based on underlying geometry, leading to better generalization to unseen illuminations and poses. In our experiments, we demonstrate the efficacy of our illumination feature representations, outperforming baseline approaches. We also show that our approach can photorealistically relight two interacting hands at real-time speeds. https://sh8.io/#/relightable_hands