CTRL-GS: Cascaded Temporal Residue Learning for 4D Gaussian Splatting
This work addresses dynamic scene reconstruction for computer vision and graphics applications, representing an incremental advancement over existing Gaussian Splatting methods.
The paper tackles novel-view synthesis for dynamic scenes by extending Gaussian Splatting to 4D with a hierarchical residual learning approach, achieving state-of-the-art visual quality and real-time rendering, with significant improvements on complex scenes involving large movements and occlusions.
Recently, Gaussian Splatting methods have emerged as a desirable substitute for prior Radiance Field methods for novel-view synthesis of scenes captured with multi-view images or videos. In this work, we propose a novel extension to 4D Gaussian Splatting for dynamic scenes. Drawing on ideas from residual learning, we hierarchically decompose the dynamic scene into a "video-segment-frame" structure, with segments dynamically adjusted by optical flow. Then, instead of directly predicting the time-dependent signals, we model the signal as the sum of video-constant values, segment-constant values, and frame-specific residuals, as inspired by the success of residual learning. This approach allows more flexible models that adapt to highly variable scenes. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets, with the greatest improvements on complex scenes with large movements, occlusions, and fine details, where current methods degrade most.