HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling
This addresses the problem of balancing quality, speed, and memory in volumetric video rendering for applications like VR/AR, representing a strong specific gain rather than a foundational breakthrough.
The paper tackled the challenge of achieving real-time, high-fidelity 6-DoF video rendering with small memory requirements, and introduced HyperReel, which achieved up to 18 frames-per-second at megapixel resolution with superior visual quality compared to existing methods.
Volumetric scene representations enable photorealistic view synthesis for static scenes and form the basis of several existing 6-DoF video techniques. However, the volume rendering procedures that drive these representations necessitate careful trade-offs in terms of quality, rendering speed, and memory efficiency. In particular, existing methods fail to simultaneously achieve real-time performance, small memory footprint, and high-quality rendering for challenging real-world scenes. To address these issues, we present HyperReel -- a novel 6-DoF video representation. The two core components of HyperReel are: (1) a ray-conditioned sample prediction network that enables high-fidelity, high frame rate rendering at high resolutions and (2) a compact and memory-efficient dynamic volume representation. Our 6-DoF video pipeline achieves the best performance compared to prior and contemporary approaches in terms of visual quality with small memory requirements, while also rendering at up to 18 frames-per-second at megapixel resolution without any custom CUDA code.