InteractAvatar: Modeling Hand-Face Interaction in Photorealistic Avatars with Deformable Gaussians
This addresses the need for realistic avatar behavior in industries such as teleconferencing, gaming, and AR/VR, though it is incremental by focusing on a specific interaction aspect.
The paper tackled the problem of modeling photorealistic hand and hand-face interactions in digital avatars, achieving high-fidelity reconstruction from monocular or multiview videos with dynamic details like wrinkles and shadows.
With the rising interest from the community in digital avatars coupled with the importance of expressions and gestures in communication, modeling natural avatar behavior remains an important challenge across many industries such as teleconferencing, gaming, and AR/VR. Human hands are the primary tool for interacting with the environment and essential for realistic human behavior modeling, yet existing 3D hand and head avatar models often overlook the crucial aspect of hand-body interactions, such as between hand and face. We present InteracttAvatar, the first model to faithfully capture the photorealistic appearance of dynamic hand and non-rigid hand-face interactions. Our novel Dynamic Gaussian Hand model, combining template model and 3D Gaussian Splatting as well as a dynamic refinement module, captures pose-dependent change, e.g. the fine wrinkles and complex shadows that occur during articulation. Importantly, our hand-face interaction module models the subtle geometry and appearance dynamics that underlie common gestures. Through experiments of novel view synthesis, self reenactment and cross-identity reenactment, we demonstrate that InteracttAvatar can reconstruct hand and hand-face interactions from monocular or multiview videos with high-fidelity details and be animated with novel poses.