Hybrid Explicit Representation for Ultra-Realistic Head Avatars
This addresses the need for high-fidelity digital avatars in applications like virtual reality or gaming, though it appears incremental as it builds on existing primitive-based rendering techniques.
The paper tackles the problem of creating ultra-realistic head avatars by introducing a hybrid explicit representation that combines UV-mapped 3D mesh and 3D Gaussian Splatting, achieving real-time rendering at >30fps at 2048x1334 resolution and exceeding state-of-the-art results.
We introduce a novel approach to creating ultra-realistic head avatars and rendering them in real-time (>30fps at $2048 \times 1334$ resolution). First, we propose a hybrid explicit representation that combines the advantages of two primitive-based efficient rendering techniques. UV-mapped 3D mesh is utilized to capture sharp and rich textures on smooth surfaces, while 3D Gaussian Splatting is employed to represent complex geometric structures. In the pipeline of modeling an avatar, after tracking parametric models based on captured multi-view RGB videos, our goal is to simultaneously optimize the texture and opacity map of mesh, as well as a set of 3D Gaussian splats localized and rigged onto the mesh facets. Specifically, we perform $α$-blending on the color and opacity values based on the merged and re-ordered z-buffer from the rasterization results of mesh and 3DGS. This process involves the mesh and 3DGS adaptively fitting the captured visual information to outline a high-fidelity digital avatar. To avoid artifacts caused by Gaussian splats crossing the mesh facets, we design a stable hybrid depth sorting strategy. Experiments illustrate that our modeled results exceed those of state-of-the-art approaches.