Adaptive Anchor Policies for Efficient 4D Gaussian Streaming
This work addresses computational inefficiency in real-time rendering and free-viewpoint video for dynamic scenes, offering an incremental improvement over existing methods.
The paper tackles the problem of inefficient anchor selection in dynamic scene reconstruction with Gaussian Splatting, which over-allocates computation, by proposing Efficient Gaussian Streaming (EGS), a reinforcement-learned policy that improves the quality-efficiency trade-off. Results show that at 256 anchors (32× fewer than baseline), EGS improves PSNR by +0.52–0.61 dB while running 1.29–1.35× faster on unseen data.
Dynamic scene reconstruction with Gaussian Splatting has enabled efficient streaming for real-time rendering and free-viewpoint video. However, most pipelines rely on fixed anchor selection such as Farthest Point Sampling (FPS), typically using 8,192 anchors regardless of scene complexity, which over-allocates computation under strict budgets. We propose Efficient Gaussian Streaming (EGS), a plug-in, budget-aware anchor sampler that replaces FPS with a reinforcement-learned policy while keeping the Gaussian streaming reconstruction backbone unchanged. The policy jointly selects an anchor budget and a subset of informative anchors under discrete constraints, balancing reconstruction quality and runtime using spatial features of the Gaussian representation. We evaluate EGS in two settings: fast rendering, which prioritizes runtime efficiency, and high-quality refinement, which enables additional optimization. Experiments on dynamic multi-view datasets show consistent improvements in the quality--efficiency trade-off over FPS sampling. On unseen data, in fast rendering at 256 anchors ($32\times$ fewer than 8,192), EGS improves PSNR by $+0.52$--$0.61$\,dB while running $1.29$--$1.35\times$ faster than IGS@8192 (N3DV and MeetingRoom). In high-quality refinement, EGS remains competitive with the full-anchor baseline at substantially lower anchor budgets. \emph{Code and pretrained checkpoints will be released upon acceptance.} \keywords{4D Gaussian Splatting \and 4D Gaussian Streaming \and Reinforcement Learning}