GauFRe: Gaussian Deformation Fields for Real-time Dynamic Novel View Synthesis
This addresses the problem of real-time dynamic novel view synthesis for applications like VR/AR, though it appears incremental as it builds on existing Gaussian-based methods.
The paper tackles dynamic scene reconstruction from monocular videos by proposing GauFRe, a method using deformable 3D Gaussians with forward-warping deformation, achieving competitive rendering quality and higher efficiency than previous state-of-the-art methods, with training in ~20 mins and 96 FPS real-time rendering on an RTX 3090 GPU.
We propose a method that achieves state-of-the-art rendering quality and efficiency on monocular dynamic scene reconstruction using deformable 3D Gaussians. Implicit deformable representations commonly model motion with a canonical space and time-dependent backward-warping deformation field. Our method, GauFRe, uses a forward-warping deformation to explicitly model non-rigid transformations of scene geometry. Specifically, we propose a template set of 3D Gaussians residing in a canonical space, and a time-dependent forward-warping deformation field to model dynamic objects. Additionally, we tailor a 3D Gaussian-specific static component supported by an inductive bias-aware initialization approach which allows the deformation field to focus on moving scene regions, improving the rendering of complex real-world motion. The differentiable pipeline is optimized end-to-end with a self-supervised rendering loss. Experiments show our method achieves competitive results and higher efficiency than both previous state-of-the-art NeRF and Gaussian-based methods. For real-world scenes, GauFRe can train in ~20 mins and offer 96 FPS real-time rendering on an RTX 3090 GPU. Project website: https://lynl7130.github.io/gaufre/index.html