Learning to Remember More with Less Memorization
This addresses memory efficiency for researchers working on long-term sequential learning with neural networks, though it appears incremental as it modifies existing memory access schemes.
The paper tackles inefficient memory utilization in RAM-like memory-augmented neural networks by proposing Uniform Writing and Cached Uniform Writing methods, which optimize information storage bounds and achieve state-of-the-art results in sequential modeling tasks.
Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning. Current RAM-like memory models maintain memory accessing every timesteps, thus they do not effectively leverage the short-term memory held in the controller. We hypothesize that this scheme of writing is suboptimal in memory utilization and introduces redundant computation. To validate our hypothesis, we derive a theoretical bound on the amount of information stored in a RAM-like system and formulate an optimization problem that maximizes the bound. The proposed solution dubbed Uniform Writing is proved to be optimal under the assumption of equal timestep contributions. To relax this assumption, we introduce modifications to the original solution, resulting in a solution termed Cached Uniform Writing. This method aims to balance between maximizing memorization and forgetting via overwriting mechanisms. Through an extensive set of experiments, we empirically demonstrate the advantages of our solutions over other recurrent architectures, claiming the state-of-the-arts in various sequential modeling tasks.