Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons
This work addresses a specific bottleneck in spiking neural networks for on-board autonomous decision-making, but it is incremental as it builds on prior ternary neuron models.
The authors tackled the problem of ternary spiking neurons performing worse than binary ones in deep Q-learning by proposing a new ternary neuron model to reduce gradient estimation bias, which improved performance in Atari games compared to existing binary neurons.
We propose a new ternary spiking neuron model to improve the representation capacity of binary spiking neurons in deep Q-learning. Although a ternary neuron model has recently been introduced to overcome the limited representation capacity offered by the binary spiking neurons, we show that its performance is worse than that of binary models in deep Q-learning tasks. We hypothesize gradient estimation bias during the training process as the underlying potential cause through mathematical and empirical analysis. We propose a novel ternary spiking neuron model to mitigate this issue by reducing the estimation bias. We use the proposed ternary spiking neuron as the fundamental computing unit in a deep spiking Q-learning network (DSQN) and evaluate the network's performance in seven Atari games from the Gym environment. Results show that the proposed ternary spiking neuron mitigates the drastic performance degradation of ternary neurons in Q-learning tasks and improves the network performance compared to the existing binary neurons, making DSQN a more practical solution for on-board autonomous decision-making tasks.