Differentiable Event Stream Simulator for Non-Rigid 3D Tracking
It addresses 3D tracking for deformable objects like human hands using event cameras, offering a novel approach without reliance on large datasets, which is incremental in advancing event-based vision.
This paper tackles the problem of non-rigid 3D tracking of deformable objects from event streams by introducing the first differentiable simulator of event streams, enabling analysis-by-synthesis without explicit event trajectories or large datasets. The method outperforms competing RGB-based methods in terms of 3D errors in experiments.
This paper introduces the first differentiable simulator of event streams, i.e., streams of asynchronous brightness change signals recorded by event cameras. Our differentiable simulator enables non-rigid 3D tracking of deformable objects (such as human hands, isometric surfaces and general watertight meshes) from event streams by leveraging an analysis-by-synthesis principle. So far, event-based tracking and reconstruction of non-rigid objects in 3D, like hands and body, has been either tackled using explicit event trajectories or large-scale datasets. In contrast, our method does not require any such processing or data, and can be readily applied to incoming event streams. We show the effectiveness of our approach for various types of non-rigid objects and compare to existing methods for non-rigid 3D tracking. In our experiments, the proposed energy-based formulations outperform competing RGB-based methods in terms of 3D errors. The source code and the new data are publicly available.