E3D: Event-Based 3D Shape Reconstruction
This work provides a more power, latency, and data-efficient 3D reconstruction method for augmented/virtual reality applications, particularly beneficial for deployment on edge devices where existing solutions are too demanding.
The paper addresses the challenge of 3D shape reconstruction on edge devices by proposing an event-based monocular approach. It tackles the problem as multi-view shape from silhouette, utilizing an event-to-silhouette neural network and a 3D differentiable renderer to achieve 3D mesh consistency.
3D shape reconstruction is a primary component of augmented/virtual reality. Despite being highly advanced, existing solutions based on RGB, RGB-D and Lidar sensors are power and data intensive, which introduces challenges for deployment in edge devices. We approach 3D reconstruction with an event camera, a sensor with significantly lower power, latency and data expense while enabling high dynamic range. While previous event-based 3D reconstruction methods are primarily based on stereo vision, we cast the problem as multi-view shape from silhouette using a monocular event camera. The output from a moving event camera is a sparse point set of space-time gradients, largely sketching scene/object edges and contours. We first introduce an event-to-silhouette (E2S) neural network module to transform a stack of event frames to the corresponding silhouettes, with additional neural branches for camera pose regression. Second, we introduce E3D, which employs a 3D differentiable renderer (PyTorch3D) to enforce cross-view 3D mesh consistency and fine-tune the E2S and pose network. Lastly, we introduce a 3D-to-events simulation pipeline and apply it to publicly available object datasets and generate synthetic event/silhouette training pairs for supervised learning.