EvRainDrop: HyperGraph-guided Completion for Effective Frame and Event Stream Aggregation
This addresses the challenge of sparse event data processing for computer vision applications, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the undersampling problem in event cameras caused by spatial sparsity by proposing a hypergraph-guided spatio-temporal event stream completion mechanism, achieving validated effectiveness in single- and multi-label event classification tasks.
Event cameras produce asynchronous event streams that are spatially sparse yet temporally dense. Mainstream event representation learning algorithms typically use event frames, voxels, or tensors as input. Although these approaches have achieved notable progress, they struggle to address the undersampling problem caused by spatial sparsity. In this paper, we propose a novel hypergraph-guided spatio-temporal event stream completion mechanism, which connects event tokens across different times and spatial locations via hypergraphs and leverages contextual information message passing to complete these sparse events. The proposed method can flexibly incorporate RGB tokens as nodes in the hypergraph within this completion framework, enabling multi-modal hypergraph-based information completion. Subsequently, we aggregate hypergraph node information across different time steps through self-attention, enabling effective learning and fusion of multi-modal features. Extensive experiments on both single- and multi-label event classification tasks fully validated the effectiveness of our proposed framework. The source code of this paper will be released on https://github.com/Event-AHU/EvRainDrop.