A Diagonal Structured State Space Model on Loihi 2 for Efficient Streaming Sequence Processing
This enables efficient real-time streaming applications of SSMs for domains requiring low-latency sequence processing, though it is incremental as it adapts an existing method to new hardware.
The paper tackled the inefficiency of token-by-token inference for deep State-Space Models (SSMs) on GPUs by implementing the SSM S4D on Intel's Loihi 2 neuromorphic processor, achieving 1000 times less energy consumption, 75 times lower latency, and 75 times higher throughput compared to a recurrent implementation on Jetson Orin Nano during token-by-token processing.
Deep State-Space Models (SSM) demonstrate state-of-the art performance on long-range sequence modeling tasks. While the recurrent structure of SSMs can be efficiently implemented as a convolution or as a parallel scan during training, recurrent token-by-token processing cannot currently be implemented efficiently on GPUs. Here, we demonstrate efficient token-by-token inference of the SSM S4D on Intel's Loihi 2 state-of-the-art neuromorphic processor. We compare this first ever neuromorphic-hardware implementation of an SSM on sMNIST, psMNIST, and sCIFAR to a recurrent and a convolutional implementation of S4D on Jetson Orin Nano (Jetson). While we find Jetson to perform better in an offline sample-by-sample based batched processing mode, Loihi 2 outperforms during token-by-token based processing, where it consumes 1000 times less energy with a 75 times lower latency and a 75 times higher throughput compared to the recurrent implementation of S4D on Jetson. This opens up new avenues towards efficient real-time streaming applications of SSMs.