Low Latency Point Cloud Rendering with Learned Splatting
This addresses the problem of visual discomfort in applications like VR/AR by enabling interactive, high-fidelity rendering for users, though it is incremental as it builds on existing splatting and neural methods.
The paper tackles the challenge of high-quality, low-latency point cloud rendering by proposing a framework that uses a neural network to estimate 3D elliptical Gaussians and differentiable splatting, achieving real-time performance with superior visual quality and generalizability.
Point cloud is a critical 3D representation with many emerging applications. Because of the point sparsity and irregularity, high-quality rendering of point clouds is challenging and often requires complex computations to recover the continuous surface representation. On the other hand, to avoid visual discomfort, the motion-to-photon latency has to be very short, under 10 ms. Existing rendering solutions lack in either quality or speed. To tackle these challenges, we present a framework that unlocks interactive, free-viewing and high-fidelity point cloud rendering. We train a generic neural network to estimate 3D elliptical Gaussians from arbitrary point clouds and use differentiable surface splatting to render smooth texture and surface normal for arbitrary views. Our approach does not require per-scene optimization, and enable real-time rendering of dynamic point cloud. Experimental results demonstrate the proposed solution enjoys superior visual quality and speed, as well as generalizability to different scene content and robustness to compression artifacts. The code is available at https://github.com/huzi96/gaussian-pcloud-render .