State Space Models for Event Cameras
This addresses a specific challenge in event-based vision for robotics or real-time applications, but it is incremental as it adapts existing SSMs to a new domain.
The paper tackles the problem of poor generalizability of deep neural networks for event-camera data when deployed at higher inference frequencies than trained on, by introducing state-space models (SSMs) with learnable timescale parameters, resulting in SSMs training 33% faster and showing minimal performance degradation (3.76 mAP drop) compared to traditional models (over 20 mAP drop).
Today, state-of-the-art deep neural networks that process event-camera data first convert a temporal window of events into dense, grid-like input representations. As such, they exhibit poor generalizability when deployed at higher inference frequencies (i.e., smaller temporal windows) than the ones they were trained on. We address this challenge by introducing state-space models (SSMs) with learnable timescale parameters to event-based vision. This design adapts to varying frequencies without the need to retrain the network at different frequencies. Additionally, we investigate two strategies to counteract aliasing effects when deploying the model at higher frequencies. We comprehensively evaluate our approach against existing methods based on RNN and Transformer architectures across various benchmarks, including Gen1 and 1 Mpx event camera datasets. Our results demonstrate that SSM-based models train 33% faster and also exhibit minimal performance degradation when tested at higher frequencies than the training input. Traditional RNN and Transformer models exhibit performance drops of more than 20 mAP, with SSMs having a drop of 3.76 mAP, highlighting the effectiveness of SSMs in event-based vision tasks.