Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers
This addresses the need for better modularity and information integration in deep learning, though it appears incremental as it builds on existing concepts like attention and feedback.
The authors tackled the problem of feed-forward neural networks processing entire hidden states rather than relevant parts, proposing Neural Function Modules (NFM) to introduce modularity with sparse arguments. The result is a flexible algorithm that improves performance in classification, out-of-domain generalization, generative modeling, and reinforcement learning.
Feed-forward neural networks consist of a sequence of layers, in which each layer performs some processing on the information from the previous layer. A downside to this approach is that each layer (or module, as multiple modules can operate in parallel) is tasked with processing the entire hidden state, rather than a particular part of the state which is most relevant for that module. Methods which only operate on a small number of input variables are an essential part of most programming languages, and they allow for improved modularity and code re-usability. Our proposed method, Neural Function Modules (NFM), aims to introduce the same structural capability into deep learning. Most of the work in the context of feed-forward networks combining top-down and bottom-up feedback is limited to classification problems. The key contribution of our work is to combine attention, sparsity, top-down and bottom-up feedback, in a flexible algorithm which, as we show, improves the results in standard classification, out-of-domain generalization, generative modeling, and learning representations in the context of reinforcement learning.