Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory
This work addresses memory allocation in neural networks for researchers in machine learning and AI, offering an incremental improvement over the Kanerva Machine.
The paper tackles the problem of bridging episodic and semantic memory in hierarchical latent variable models by introducing a differentiable, locally block allocated latent memory scheme, which achieves state-of-the-art conditional likelihood values on binarized MNIST (≤41.58 nats/image) and binarized Omniglot (≤66.24 nats/image), with competitive performance on other datasets.
Episodic and semantic memory are critical components of the human memory model. The theory of complementary learning systems (McClelland et al., 1995) suggests that the compressed representation produced by a serial event (episodic memory) is later restructured to build a more generalized form of reusable knowledge (semantic memory). In this work we develop a new principled Bayesian memory allocation scheme that bridges the gap between episodic and semantic memory via a hierarchical latent variable model. We take inspiration from traditional heap allocation and extend the idea of locally contiguous memory to the Kanerva Machine, enabling a novel differentiable block allocated latent memory. In contrast to the Kanerva Machine, we simplify the process of memory writing by treating it as a fully feed forward deterministic process, relying on the stochasticity of the read key distribution to disperse information within the memory. We demonstrate that this allocation scheme improves performance in memory conditional image generation, resulting in new state-of-the-art conditional likelihood values on binarized MNIST (<=41.58 nats/image) , binarized Omniglot (<=66.24 nats/image), as well as presenting competitive performance on CIFAR10, DMLab Mazes, Celeb-A and ImageNet32x32.