Human-Level Control through Directly-Trained Deep Spiking Q-Networks
This work advances energy-efficient AI for neuromorphic hardware by enabling direct training of spiking neural networks in reinforcement learning, though it is incremental as it builds on existing DQN methods.
The paper tackles the challenge of Deep Spiking Reinforcement Learning (DSRL) by proposing a Deep Spiking Q-Network (DSQN) that directly trains spiking neural networks, achieving state-of-the-art performance on 17 Atari games with advantages in performance, stability, robustness, and energy-efficiency.
As the third-generation neural networks, Spiking Neural Networks (SNNs) have great potential on neuromorphic hardware because of their high energy-efficiency. However, Deep Spiking Reinforcement Learning (DSRL), i.e., the Reinforcement Learning (RL) based on SNNs, is still in its preliminary stage due to the binary output and the non-differentiable property of the spiking function. To address these issues, we propose a Deep Spiking Q-Network (DSQN) in this paper. Specifically, we propose a directly-trained deep spiking reinforcement learning architecture based on the Leaky Integrate-and-Fire (LIF) neurons and Deep Q-Network (DQN). Then, we adapt a direct spiking learning algorithm for the Deep Spiking Q-Network. We further demonstrate the advantages of using LIF neurons in DSQN theoretically. Comprehensive experiments have been conducted on 17 top-performing Atari games to compare our method with the state-of-the-art conversion method. The experimental results demonstrate the superiority of our method in terms of performance, stability, robustness and energy-efficiency. To the best of our knowledge, our work is the first one to achieve state-of-the-art performance on multiple Atari games with the directly-trained SNN.