Language and Experience: A Computational Model of Social Learning in Complex Tasks
This addresses the challenge of enabling AI systems and humans to combine language and experience for more efficient learning in complex tasks, with incremental improvements in modeling social learning.
The paper tackles the problem of how to integrate linguistic guidance with direct experience for safe and rapid learning in new environments, showing that their computational framework accelerates learning by reducing risky interactions and speeding up discoveries in 10 video games.
The ability to combine linguistic guidance from others with direct experience is central to human development, enabling safe and rapid learning in new environments. How do people integrate these two sources of knowledge, and how might AI systems? We present a computational framework that models social learning as joint probabilistic inference over structured, executable world models given sensorimotor and linguistic data. We make this possible by turning a pretrained language model into a probabilistic model of how humans share advice conditioned on their beliefs, allowing our agents both to generate advice for others and to interpret linguistic input as evidence during Bayesian inference. Using behavioral experiments and simulations across 10 video games, we show how linguistic guidance can shape exploration and accelerate learning by reducing risky interactions and speeding up key discoveries in both humans and models. We further explore how knowledge can accumulate across generations through iterated learning experiments and demonstrate successful knowledge transfer between humans and models -- revealing how structured, language-compatible representations might enable human-machine collaborative learning.