Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
This work addresses the gap between artificial neural network architectures and biological models for researchers in machine learning and neuroscience, but it is incremental as it builds on existing ResNet and RNN frameworks.
The paper tackles the problem of connecting residual networks, recurrent neural networks, and visual cortex models by showing that a shallow RNN is equivalent to a deep ResNet with weight sharing, achieving similar performance with far fewer parameters, and tests this on CIFAR-10 and ImageNet datasets.
We discuss relations between Residual Networks (ResNet), Recurrent Neural Networks (RNNs) and the primate visual cortex. We begin with the observation that a special type of shallow RNN is exactly equivalent to a very deep ResNet with weight sharing among the layers. A direct implementation of such a RNN, although having orders of magnitude fewer parameters, leads to a performance similar to the corresponding ResNet. We propose 1) a generalization of both RNN and ResNet architectures and 2) the conjecture that a class of moderately deep RNNs is a biologically-plausible model of the ventral stream in visual cortex. We demonstrate the effectiveness of the architectures by testing them on the CIFAR-10 and ImageNet dataset.