Memory Mosaics
This addresses the problem of achieving transparent compositional learning in AI, though it appears incremental as it builds on transformer-like capabilities.
The paper introduces Memory Mosaics, networks of associative memories that tackle prediction tasks with compositional and in-context learning capabilities, achieving performance comparable to or better than transformers on medium-scale language modeling.
Memory Mosaics are networks of associative memories working in concert to achieve a prediction task of interest. Like transformers, memory mosaics possess compositional capabilities and in-context learning capabilities. Unlike transformers, memory mosaics achieve these capabilities in comparatively transparent way ("predictive disentanglement"). We illustrate these capabilities on a toy example and also show that memory mosaics perform as well or better than transformers on medium-scale language modeling tasks.