Monocular Dynamic View Synthesis: A Reality Check
This work addresses the gap between experimental protocols and real-world capture for researchers in computer vision, highlighting incremental improvements in evaluation.
The paper tackles the problem of dynamic view synthesis from monocular video by revealing that existing methods rely on leaked multi-view signals during training, and shows that state-of-the-art approaches experience a 1-2 dB drop in performance without these cues and a 4-5 dB drop for complex motion.
We study the recent progress on dynamic view synthesis (DVS) from monocular video. Though existing approaches have demonstrated impressive results, we show a discrepancy between the practical capture process and the existing experimental protocols, which effectively leaks in multi-view signals during training. We define effective multi-view factors (EMFs) to quantify the amount of multi-view signal present in the input capture sequence based on the relative camera-scene motion. We introduce two new metrics: co-visibility masked image metrics and correspondence accuracy, which overcome the issue in existing protocols. We also propose a new iPhone dataset that includes more diverse real-life deformation sequences. Using our proposed experimental protocol, we show that the state-of-the-art approaches observe a 1-2 dB drop in masked PSNR in the absence of multi-view cues and 4-5 dB drop when modeling complex motion. Code and data can be found at https://hangg7.com/dycheck.