Geometry-guided Online 3D Video Synthesis with Multi-View Temporal Consistency
This work addresses the challenge of efficient online 3D video synthesis for applications like virtual reality or film production, though it is incremental as it builds on existing image-based rendering techniques.
The paper tackles the problem of achieving high-quality, consistent novel-view video synthesis from sparse multi-view inputs by introducing a geometry-guided method that uses refined depth maps and a pre-trained blending network to reduce flickering artifacts and ensure temporal consistency.
We introduce a novel geometry-guided online video view synthesis method with enhanced view and temporal consistency. Traditional approaches achieve high-quality synthesis from dense multi-view camera setups but require significant computational resources. In contrast, selective-input methods reduce this cost but often compromise quality, leading to multi-view and temporal inconsistencies such as flickering artifacts. Our method addresses this challenge to deliver efficient, high-quality novel-view synthesis with view and temporal consistency. The key innovation of our approach lies in using global geometry to guide an image-based rendering pipeline. To accomplish this, we progressively refine depth maps using color difference masks across time. These depth maps are then accumulated through truncated signed distance fields in the synthesized view's image space. This depth representation is view and temporally consistent, and is used to guide a pre-trained blending network that fuses multiple forward-rendered input-view images. Thus, the network is encouraged to output geometrically consistent synthesis results across multiple views and time. Our approach achieves consistent, high-quality video synthesis, while running efficiently in an online manner.