Encoding Event-Based Data With a Hybrid SNN Guided Variational Auto-encoder in Neuromorphic Hardware
This enables real-time learning from real-world event data for neuromorphic systems, though it is incremental as it builds on existing VAE and SNN methods.
The authors tackled the problem of learning from unlabeled event-based data in neuromorphic hardware by proposing a Hybrid Guided Variational Autoencoder (VAE) that encodes data from a Dynamic Vision Sensor (DVS) into a latent space using a Spiking Neural Network (SNN), achieving 87% classification accuracy on the DVSGesture dataset.
Neuromorphic hardware equipped with learning capabilities can adapt to new, real-time data. While models of Spiking Neural Networks (SNNs) can now be trained using gradient descent to reach an accuracy comparable to equivalent conventional neural networks, such learning often relies on external labels. However, real-world data is unlabeled which can make supervised methods inapplicable. To solve this problem, we propose a Hybrid Guided Variational Autoencoder (VAE) which encodes event based data sensed by a Dynamic Vision Sensor (DVS) into a latent space representation using an SNN. These representations can be used as an embedding to measure data similarity and predict labels in real-world data. We show that the Hybrid Guided-VAE achieves 87% classification accuracy on the DVSGesture dataset and it can encode the sparse, noisy inputs into an interpretable latent space representation, visualized through T-SNE plots. We also implement the encoder component of the model on neuromorphic hardware and discuss the potential for our algorithm to enable real-time learning from real-world event data.