NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos
It addresses the problem of dynamic 3D scene understanding for applications in computer vision and robotics, representing an incremental advance by extending neural radiance fields to include velocity fields.
The paper tackles modeling 3D scene dynamics from multi-view videos by learning geometry, appearance, and physical velocity, enabling applications like future frame extrapolation and unsupervised 3D semantic scene decomposition, with results showing superior performance over baselines in these tasks.
In this paper, we aim to model 3D scene dynamics from multi-view videos. Unlike the majority of existing works which usually focus on the common task of novel view synthesis within the training time period, we propose to simultaneously learn the geometry, appearance, and physical velocity of 3D scenes only from video frames, such that multiple desirable applications can be supported, including future frame extrapolation, unsupervised 3D semantic scene decomposition, and dynamic motion transfer. Our method consists of three major components, 1) the keyframe dynamic radiance field, 2) the interframe velocity field, and 3) a joint keyframe and interframe optimization module which is the core of our framework to effectively train both networks. To validate our method, we further introduce two dynamic 3D datasets: 1) Dynamic Object dataset, and 2) Dynamic Indoor Scene dataset. We conduct extensive experiments on multiple datasets, demonstrating the superior performance of our method over all baselines, particularly in the critical tasks of future frame extrapolation and unsupervised 3D semantic scene decomposition.