Taming the ReLU with Parallel Dither in a Deep Neural Network
This addresses a specific issue in deep neural networks for researchers and practitioners, but appears incremental as it builds on existing ReLU usage.
The paper tackles the problem of ReLU activation functions causing nonlinear distortion products (decoy features) that lead to overfitting, and shows that using Parallel Dither suppresses these decoys, enabling rapid and reliable learning.
Rectified Linear Units (ReLU) seem to have displaced traditional 'smooth' nonlinearities as activation-function-du-jour in many - but not all - deep neural network (DNN) applications. However, nobody seems to know why. In this article, we argue that ReLU are useful because they are ideal demodulators - this helps them perform fast abstract learning. However, this fast learning comes at the expense of serious nonlinear distortion products - decoy features. We show that Parallel Dither acts to suppress the decoy features, preventing overfitting and leaving the true features cleanly demodulated for rapid, reliable learning.