Lightweight High-Fidelity Low-Bitrate Talking Face Compression for 3D Video Conference
This work addresses the problem of efficient 3D video conferencing for users needing immersive communication, representing an incremental improvement by combining existing techniques like FLAME and 3DGS with novel compression schemes.
The paper tackles the challenge of achieving high-fidelity 3D talking face representation at low bitrates for immersive communication by proposing a lightweight framework that integrates FLAME-based parametric modeling with 3DGS neural rendering, resulting in superior rate-distortion performance and high-quality facial rendering at extremely low bitrates.
The demand for immersive and interactive communication has driven advancements in 3D video conferencing, yet achieving high-fidelity 3D talking face representation at low bitrates remains a challenge. Traditional 2D video compression techniques fail to preserve fine-grained geometric and appearance details, while implicit neural rendering methods like NeRF suffer from prohibitive computational costs. To address these challenges, we propose a lightweight, high-fidelity, low-bitrate 3D talking face compression framework that integrates FLAME-based parametric modeling with 3DGS neural rendering. Our approach transmits only essential facial metadata in real time, enabling efficient reconstruction with a Gaussian-based head model. Additionally, we introduce a compact representation and compression scheme, including Gaussian attribute compression and MLP optimization, to enhance transmission efficiency. Experimental results demonstrate that our method achieves superior rate-distortion performance, delivering high-quality facial rendering at extremely low bitrates, making it well-suited for real-time 3D video conferencing applications.