EvDNeRF: Reconstructing Event Data with Dynamic Neural Radiance Fields
This work addresses the challenge of simulating event data for dynamic scenes, which is incremental as it extends neural radiance fields to event-based data for both rigid and non-rigid deformations.
The paper tackles the problem of reconstructing event data for dynamic scenes with fast motion, which standard cameras cannot capture, by introducing EvDNeRF, a pipeline that predicts eventstreams from any viewpoint and timestamp, outperforming baselines in test-time predictions at fine time resolutions.
We present EvDNeRF, a pipeline for generating event data and training an event-based dynamic NeRF, for the purpose of faithfully reconstructing eventstreams on scenes with rigid and non-rigid deformations that may be too fast to capture with a standard camera. Event cameras register asynchronous per-pixel brightness changes at MHz rates with high dynamic range, making them ideal for observing fast motion with almost no motion blur. Neural radiance fields (NeRFs) offer visual-quality geometric-based learnable rendering, but prior work with events has only considered reconstruction of static scenes. Our EvDNeRF can predict eventstreams of dynamic scenes from a static or moving viewpoint between any desired timestamps, thereby allowing it to be used as an event-based simulator for a given scene. We show that by training on varied batch sizes of events, we can improve test-time predictions of events at fine time resolutions, outperforming baselines that pair standard dynamic NeRFs with event generators. We release our simulated and real datasets, as well as code for multi-view event-based data generation and the training and evaluation of EvDNeRF models (https://github.com/anish-bhattacharya/EvDNeRF).