SuperEvent: Cross-Modal Learning of Event-based Keypoint Detection for SLAM
This addresses the challenge of integrating event sensors into SLAM systems for robotics or autonomous vehicles, though it is incremental as it builds on existing frame-based methods.
The paper tackles the problem of event-based keypoint detection and matching for SLAM, which suffers from motion-dependent appearance and noise, by proposing SuperEvent, a data-driven method that uses self-supervision from frame-based detectors and a novel event representation, resulting in surpassing state-of-the-art event-based SLAM by a wide margin.
Event-based keypoint detection and matching holds significant potential, enabling the integration of event sensors into highly optimized Visual SLAM systems developed for frame cameras over decades of research. Unfortunately, existing approaches struggle with the motion-dependent appearance of keypoints and the complex noise prevalent in event streams, resulting in severely limited feature matching capabilities and poor performance on downstream tasks. To mitigate this problem, we propose SuperEvent, a data-driven approach to predict stable keypoints with expressive descriptors. Due to the absence of event datasets with ground truth keypoint labels, we leverage existing frame-based keypoint detectors on readily available event-aligned and synchronized gray-scale frames for self-supervision: we generate temporally sparse keypoint pseudo-labels considering that events are a product of both scene appearance and camera motion. Combined with our novel, information-rich event representation, we enable SuperEvent to effectively learn robust keypoint detection and description in event streams. Finally, we demonstrate the usefulness of SuperEvent by its integration into a modern sparse keypoint and descriptor-based SLAM framework originally developed for traditional cameras, surpassing the state-of-the-art in event-based SLAM by a wide margin. Source code is available at https://ethz-mrl.github.io/SuperEvent/.