Use of recurrent infomax to improve the memory capability of input-driven recurrent neural networks
This work addresses improving memory capabilities in RNNs for applications like temporal data processing, but it appears incremental as it builds on existing RI methods without major breakthroughs.
The study tackled the problem of varying information processing capabilities in input-driven recurrent neural networks (RNNs) by analyzing RNNs optimized with recurrent infomax (RI), an unsupervised learning scheme that maximizes mutual information. The result showed that RI leads to a delay-line structure and superior short-term memory for storing temporal input information.
The inherent transient dynamics of recurrent neural networks (RNNs) have been exploited as a computational resource in input-driven RNNs. However, the information processing capability varies from RNN to RNN, depending on their properties. Many authors have investigated the dynamics of RNNs and their relevance to the information processing capability. In this study, we present a detailed analysis of the information processing capability of an RNN optimized by recurrent infomax (RI), which is an unsupervised learning scheme that maximizes the mutual information of RNNs by adjusting the connection strengths of the network. Thus, we observe that a delay-line structure emerges from the RI and the network optimized by the RI possesses superior short-term memory, which is the ability to store the temporal information of the input stream in its transient dynamics.