End-to-End Rate-Distortion Optimized 3D Gaussian Representation
This work addresses storage limitations for practical applications of 3D Gaussian representations, such as in rendering and 3D modeling, with incremental improvements over prior compression schemes.
The paper tackles the high storage overhead of 3D Gaussian Splatting by formulating it as an end-to-end rate-distortion optimization problem, resulting in a method that reduces size by over 40x and surpasses existing methods in performance.
3D Gaussian Splatting (3DGS) has become an emerging technique with remarkable potential in 3D representation and image rendering. However, the substantial storage overhead of 3DGS significantly impedes its practical applications. In this work, we formulate the compact 3D Gaussian learning as an end-to-end Rate-Distortion Optimization (RDO) problem and propose RDO-Gaussian that can achieve flexible and continuous rate control. RDO-Gaussian addresses two main issues that exist in current schemes: 1) Different from prior endeavors that minimize the rate under the fixed distortion, we introduce dynamic pruning and entropy-constrained vector quantization (ECVQ) that optimize the rate and distortion at the same time. 2) Previous works treat the colors of each Gaussian equally, while we model the colors of different regions and materials with learnable numbers of parameters. We verify our method on both real and synthetic scenes, showcasing that RDO-Gaussian greatly reduces the size of 3D Gaussian over 40x, and surpasses existing methods in rate-distortion performance.