Hadamard Representations: Augmenting Hyperbolic Tangents in RL
This addresses performance issues in RL for researchers and practitioners by improving activation functions, though it is incremental as it builds on existing methods to mitigate a known bottleneck.
The paper tackled the dying neuron problem in reinforcement learning with continuously differentiable activations like tanh, proposing a Hadamard representation that resulted in faster learning, reduced dead neurons, and increased effective rank in Atari benchmarks using DQN, PPO, and PQN.
Activation functions are one of the key components of a deep neural network. The most commonly used activation functions can be classed into the category of continuously differentiable (e.g. tanh) and piece-wise linear functions (e.g. ReLU), both having their own strengths and drawbacks with respect to downstream performance and representation capacity through learning. In reinforcement learning, the performance of continuously differentiable activations often falls short as compared to piece-wise linear functions. We show that the dying neuron problem in RL is not exclusive to ReLUs and actually leads to additional problems in the case of continuously differentiable activations such as tanh. To alleviate the dying neuron problem with these activations, we propose a Hadamard representation that unlocks the advantages of continuously differentiable activations. Using DQN, PPO and PQN in the Atari domain, we show faster learning, a reduction in dead neurons and increased effective rank.