On the Relationship Between Variational Inference and Auto-Associative Memory
This work addresses the challenge of combining inference and memory in AI systems, but it appears incremental as it builds on existing variational and neural network methods.
The authors tackled the problem of unifying perceptual inference and memory retrieval by proposing a variational inference formulation for auto-associative memories, which integrates memory-dependent priors into latent representations. They evaluated this framework on CIFAR10 and CLEVR datasets, comparing it with models like Hopfield Networks and Neural Turing Machines.
In this article, we propose a variational inference formulation of auto-associative memories, allowing us to combine perceptual inference and memory retrieval into the same mathematical framework. In this formulation, the prior probability distribution onto latent representations is made memory dependent, thus pulling the inference process towards previously stored representations. We then study how different neural network approaches to variational inference can be applied in this framework. We compare methods relying on amortized inference such as Variational Auto Encoders and methods relying on iterative inference such as Predictive Coding and suggest combining both approaches to design new auto-associative memory models. We evaluate the obtained algorithms on the CIFAR10 and CLEVR image datasets and compare them with other associative memory models such as Hopfield Networks, End-to-End Memory Networks and Neural Turing Machines.