Feedback-Gated Rectified Linear Units
This work addresses the under-explored role of feedback in artificial neural networks, offering incremental improvements for tasks like image classification.
The authors tackled the problem of improving neural network performance by introducing a biologically inspired feedback mechanism that gates rectified linear units, resulting in faster convergence, better performance, and more robustness to noise on MNIST, with less consistent benefits on CIFAR-10.
Feedback connections play a prominent role in the human brain but have not received much attention in artificial neural network research. Here, a biologically inspired feedback mechanism which gates rectified linear units is proposed. On the MNIST dataset, autoencoders with feedback show faster convergence, better performance, and more robustness to noise compared to their counterparts without feedback. Some benefits, although less pronounced and less consistent, can be observed when networks with feedback are applied on the CIFAR-10 dataset.