Opening the Black Box of 3D Reconstruction Error Analysis with VECTOR
This addresses the problem of inaccurate scientific inferences or risks in space exploration due to reconstruction errors, but it is incremental as it focuses on improving error analysis rather than the core reconstruction method.
The paper tackles the challenge of analyzing reconstruction errors in 3D scene reconstruction from 2D images, presenting VECTOR, a visual analysis tool that improves error inspection for bundle adjustment, as demonstrated in its use with the Mars 2020 mission.
Reconstruction of 3D scenes from 2D images is a technical challenge that impacts domains from Earth and planetary sciences and space exploration to augmented and virtual reality. Typically, reconstruction algorithms first identify common features across images and then minimize reconstruction errors after estimating the shape of the terrain. This bundle adjustment (BA) step optimizes around a single, simplifying scalar value that obfuscates many possible causes of reconstruction errors (e.g., initial estimate of the position and orientation of the camera, lighting conditions, ease of feature detection in the terrain). Reconstruction errors can lead to inaccurate scientific inferences or endanger a spacecraft exploring a remote environment. To address this challenge, we present VECTOR, a visual analysis tool that improves error inspection for stereo reconstruction BA. VECTOR provides analysts with previously unavailable visibility into feature locations, camera pose, and computed 3D points. VECTOR was developed in partnership with the Perseverance Mars Rover and Ingenuity Mars Helicopter terrain reconstruction team at the NASA Jet Propulsion Laboratory. We report on how this tool was used to debug and improve terrain reconstruction for the Mars 2020 mission.