Hierarchical Residuals Exploit Brain-Inspired Compositionality
This work addresses performance and efficiency issues in neural networks for researchers and practitioners, but it appears incremental as it builds on existing ResNet architectures.
The authors tackled the problem of improving deep convolutional neural networks by introducing hierarchical residual connections inspired by brain organization, resulting in a boost in accuracy and faster learning.
We present Hierarchical Residual Networks (HiResNets), deep convolutional neural networks with long-range residual connections between layers at different hierarchical levels. HiResNets draw inspiration on the organization of the mammalian brain by replicating the direct connections from subcortical areas to the entire cortical hierarchy. We show that the inclusion of hierarchical residuals in several architectures, including ResNets, results in a boost in accuracy and faster learning. A detailed analysis of our models reveals that they perform hierarchical compositionality by learning feature maps relative to the compressed representations provided by the skip connections.