Building, Reusing, and Generalizing Abstract Representations from Concrete Sequences
This work addresses memory and transfer inefficiencies in sequence learning for cognitive modeling and AI, offering a novel approach but with incremental improvements over existing methods.
The paper tackles the problem of sequence learning models lacking abstraction ability, which causes memory inefficiency and poor transfer, by introducing a non-parametric hierarchical variable learning model (HVM) that learns chunks and abstracts them as variables; it shows HVM learns a more efficient dictionary than Lempel-Ziv on babyLM datasets and correlates with human recall times in transfer tasks, unlike large language models.
Humans excel at learning abstract patterns across different sequences, filtering out irrelevant details, and transferring these generalized concepts to new sequences. In contrast, many sequence learning models lack the ability to abstract, which leads to memory inefficiency and poor transfer. We introduce a non-parametric hierarchical variable learning model (HVM) that learns chunks from sequences and abstracts contextually similar chunks as variables. HVM efficiently organizes memory while uncovering abstractions, leading to compact sequence representations. When learning on language datasets such as babyLM, HVM learns a more efficient dictionary than standard compression algorithms such as Lempel-Ziv. In a sequence recall task requiring the acquisition and transfer of variables embedded in sequences, we demonstrate HVM's sequence likelihood correlates with human recall times. In contrast, large language models (LLMs) struggle to transfer abstract variables as effectively as humans. From HVM's adjustable layer of abstraction, we demonstrate that the model realizes a precise trade-off between compression and generalization. Our work offers a cognitive model that captures the learning and transfer of abstract representations in human cognition and differentiates itself from LLMs.