Meta-Learning Neural Mechanisms rather than Bayesian Priors
This provides insights into efficient meta-learning and links symbolic theories with neural mechanisms, offering practical implications for AI development.
The study investigated what meta-learning imparts to models, finding that meta-trained models learn neural mechanisms like counters rather than simplicity-based priors, and that meta-training on a single formal language can be as effective as on 5000 languages if it incentivizes useful mechanisms.
Children acquire language despite being exposed to several orders of magnitude less data than large language models require. Meta-learning has been proposed as a way to integrate human-like learning biases into neural-network architectures, combining both the structured generalizations of symbolic models with the scalability of neural-network models. But what does meta-learning exactly imbue the model with? We investigate the meta-learning of formal languages and find that, contrary to previous claims, meta-trained models are not learning simplicity-based priors when meta-trained on datasets organised around simplicity. Rather, we find evidence that meta-training imprints neural mechanisms (such as counters) into the model, which function like cognitive primitives for the network on downstream tasks. Most surprisingly, we find that meta-training on a single formal language can provide as much improvement to a model as meta-training on 5000 different formal languages, provided that the formal language incentivizes the learning of useful neural mechanisms. Taken together, our findings provide practical implications for efficient meta-learning paradigms and new theoretical insights into linking symbolic theories and neural mechanisms.