Per-Gaussian Embedding-Based Deformation for Deformable 3D Gaussian Splatting
This work addresses the challenge of dynamic scene representation for novel view synthesis, offering an incremental improvement in accuracy for applications like virtual reality and video processing.
The paper tackles the problem of accurately reconstructing complex dynamic scenes in 3D Gaussian Splatting by proposing a deformation field based on per-Gaussian embeddings, achieving improved reconstruction quality over previous methods.
As 3D Gaussian Splatting (3DGS) provides fast and high-quality novel view synthesis, it is a natural extension to deform a canonical 3DGS to multiple frames for representing a dynamic scene. However, previous works fail to accurately reconstruct complex dynamic scenes. We attribute the failure to the design of the deformation field, which is built as a coordinate-based function. This approach is problematic because 3DGS is a mixture of multiple fields centered at the Gaussians, not just a single coordinate-based framework. To resolve this problem, we define the deformation as a function of per-Gaussian embeddings and temporal embeddings. Moreover, we decompose deformations as coarse and fine deformations to model slow and fast movements, respectively. Also, we introduce a local smoothness regularization for per-Gaussian embedding to improve the details in dynamic regions. Project page: https://jeongminb.github.io/e-d3dgs/