Event-based Camera Tracker by $\nabla$t NeRF
This addresses camera pose estimation for event-based cameras, which is an incremental advancement by combining implicit scene representation with sparse event data.
The paper tackles the problem of recovering camera pose from sparse event-based camera data by minimizing the error between observed events and the temporal gradient of a neural radiance field (NeRF), resulting in a novel framework called TeGRA that enables efficient pose tracking.
When a camera travels across a 3D world, only a fraction of pixel value changes; an event-based camera observes the change as sparse events. How can we utilize sparse events for efficient recovery of the camera pose? We show that we can recover the camera pose by minimizing the error between sparse events and the temporal gradient of the scene represented as a neural radiance field (NeRF). To enable the computation of the temporal gradient of the scene, we augment NeRF's camera pose as a time function. When the input pose to the NeRF coincides with the actual pose, the output of the temporal gradient of NeRF equals the observed intensity changes on the event's points. Using this principle, we propose an event-based camera pose tracking framework called TeGRA which realizes the pose update by using the sparse event's observation. To the best of our knowledge, this is the first camera pose estimation algorithm using the scene's implicit representation and the sparse intensity change from events.