Temporal-Spatial Processing of Event Camera Data via Delay-Loop Reservoir Neural Network
This work addresses video processing for event camera applications, offering an incremental improvement by leveraging temporal information often overlooked in neural networks.
The paper tackles the problem of processing event camera videos by proposing the Temporal-Spatial Conjecture, which suggests that temporal components carry significant information, and validates it using a Visual Markov Model and mutual information estimation, resulting in an 18% improvement in classification accuracy.
This paper describes a temporal-spatial model for video processing with special applications to processing event camera videos. We propose to study a conjecture motivated by our previous study of video processing with delay loop reservoir (DLR) neural network, which we call Temporal-Spatial Conjecture (TSC). The TSC postulates that there is significant information content carried in the temporal representation of a video signal and that machine learning algorithms would benefit from separate optimization of the spatial and temporal components for intelligent processing. To verify or refute the TSC, we propose a Visual Markov Model (VMM) which decompose the video into spatial and temporal components and estimate the mutual information (MI) of these components. Since computation of video mutual information is complex and time consuming, we use a Mutual Information Neural Network to estimate the bounds of the mutual information. Our result shows that the temporal component carries significant MI compared to that of the spatial component. This finding has often been overlooked in neural network literature. In this paper, we will exploit this new finding to guide our design of a delay-loop reservoir neural network for event camera classification, which results in a 18% improvement on classification accuracy.