Recovering Parametric Scenes from Very Few Time-of-Flight Pixels
This addresses the challenge of 3D scene reconstruction for applications like robotics or augmented reality using sparse, low-resolution sensors, though it is incremental with a focus on simple parametric scenes.
The paper tackles the problem of recovering 3D parametric scene geometry, such as object pose, from very few depth measurements using low-cost time-of-flight sensors, achieving effective pose recovery in simulations and real-world captures with as few as 15 pixels.
We aim to recover the geometry of 3D parametric scenes using very few depth measurements from low-cost, commercially available time-of-flight sensors. These sensors offer very low spatial resolution (i.e., a single pixel), but image a wide field-of-view per pixel and capture detailed time-of-flight data in the form of time-resolved photon counts. This time-of-flight data encodes rich scene information and thus enables recovery of simple scenes from sparse measurements. We investigate the feasibility of using a distributed set of few measurements (e.g., as few as 15 pixels) to recover the geometry of simple parametric scenes with a strong prior, such as estimating the 6D pose of a known object. To achieve this, we design a method that utilizes both feed-forward prediction to infer scene parameters, and differentiable rendering within an analysis-by-synthesis framework to refine the scene parameter estimate. We develop hardware prototypes and demonstrate that our method effectively recovers object pose given an untextured 3D model in both simulations and controlled real-world captures, and show promising initial results for other parametric scenes. We additionally conduct experiments to explore the limits and capabilities of our imaging solution.