E2PNet: Event to Point Cloud Registration with Spatio-Temporal Representation Learning
This addresses the registration challenge for event cameras in computer vision, enabling robust performance under extreme conditions like illumination or fast motion, but it is incremental as it adapts existing RGB-based frameworks to a new sensor type.
They tackled the problem of 2D-3D registration for event cameras, which had no prior work, by proposing E2PNet, a learning-based method that uses a novel feature representation network to encode event data into a 2D grid-shaped tensor, achieving superiority over hand-crafted and other learning-based methods on datasets like MVSEC and VECtor.
Event cameras have emerged as a promising vision sensor in recent years due to their unparalleled temporal resolution and dynamic range. While registration of 2D RGB images to 3D point clouds is a long-standing problem in computer vision, no prior work studies 2D-3D registration for event cameras. To this end, we propose E2PNet, the first learning-based method for event-to-point cloud registration. The core of E2PNet is a novel feature representation network called Event-Points-to-Tensor (EP2T), which encodes event data into a 2D grid-shaped feature tensor. This grid-shaped feature enables matured RGB-based frameworks to be easily used for event-to-point cloud registration, without changing hyper-parameters and the training procedure. EP2T treats the event input as spatio-temporal point clouds. Unlike standard 3D learning architectures that treat all dimensions of point clouds equally, the novel sampling and information aggregation modules in EP2T are designed to handle the inhomogeneity of the spatial and temporal dimensions. Experiments on the MVSEC and VECtor datasets demonstrate the superiority of E2PNet over hand-crafted and other learning-based methods. Compared to RGB-based registration, E2PNet is more robust to extreme illumination or fast motion due to the use of event data. Beyond 2D-3D registration, we also show the potential of EP2T for other vision tasks such as flow estimation, event-to-image reconstruction and object recognition. The source code can be found at: https://github.com/Xmu-qcj/E2PNet.