3D scene reconstruction from monocular spherical video with motion parallax
This enables 3D reconstruction of surrounding environments from any 360-degree video with motion parallax, applicable to film previsualization, scene synthesis, and motion capture, though it is incremental as it builds on existing spherical stereo methods.
The paper tackles 3D scene reconstruction from monocular spherical video by using motion parallax between two adjacent frames to retrieve depth information, achieving coverage on up to 97% of the sphere and estimating depths over 30 meters at speeds of 30 km/h.
In this paper, we describe a method to capture nearly entirely spherical (360 degree) depth information using two adjacent frames from a single spherical video with motion parallax. After illustrating a spherical depth information retrieval using two spherical cameras, we demonstrate monocular spherical stereo by using stabilized first-person video footage. Experiments demonstrated that the depth information was retrieved on up to 97% of the entire sphere in solid angle. At a speed of 30 km/h, we were able to estimate the depth of an object located over 30 m from the camera. We also reconstructed the 3D structures (point cloud) using the obtained depth data and confirmed the structures can be clearly observed. We can apply this method to 3D structure retrieval of surrounding environments such as 1) previsualization, location hunting/planning of a film, 2) real scene/computer graphics synthesis and 3) motion capture. Thanks to its simplicity, this method can be applied to various videos. As there is no pre-condition other than to be a 360 video with motion parallax, we can use any 360 videos including those on the Internet to reconstruct the surrounding environments. The cameras can be lightweight enough to be mounted on a drone. We also demonstrated such applications.