Differentiable Neural Computers with Memory Demon
This work addresses memory efficiency in neural networks with external memory, offering an incremental improvement for tasks requiring iterative content modification.
The paper tackles the problem of improving Differentiable Neural Computers (DNCs) by addressing information theoretic properties of memory contents, introducing a memory demon that implicitly modifies memory via additive input encoding to maximize mutual information, resulting in enhanced performance.
A Differentiable Neural Computer (DNC) is a neural network with an external memory which allows for iterative content modification via read, write and delete operations. We show that information theoretic properties of the memory contents play an important role in the performance of such architectures. We introduce a novel concept of memory demon to DNC architectures which modifies the memory contents implicitly via additive input encoding. The goal of the memory demon is to maximize the expected sum of mutual information of the consecutive external memory contents.