Sophisticated Learning: A novel algorithm for active learning during model-based planning
This work addresses decision-making under radical uncertainty for biologically inspired AI agents, representing an incremental improvement over existing methods.
The authors tackled the problem of decision-making under uncertainty by introducing Sophisticated Learning (SL), a planning-to-learn algorithm that embeds active parameter learning within the Sophisticated Inference framework, and found that in a seasonal foraging task, SL agents survived 8.2% longer than SI and 35% longer than Bayes-adaptive Reinforcement Learning in early trials, with SL reaching convergence 40% faster than SI.
We introduce Sophisticated Learning (SL), a planning-to-learn algorithm that embeds active parameter learning inside the Sophisticated Inference (SI) tree-search framework of Active Inference. Unlike SI -- which optimizes beliefs about hidden states -- SL also updates beliefs about model parameters within each simulated branch, enabling counterfactual reasoning about how future observations would improve subsequent planning. We compared SL with Bayes-adaptive Reinforcement Learning (BARL) agents as well as with its parent algorithm, SI. Using a biologically inspired seasonal foraging task in which resources shift probabilistically over a 10x10 grid, we designed experiments that forced agents to balance probabilistic reward harvesting against information gathering. In early trials, where rapid learning is vital, SL agents survive, on average, 8.2% longer than SI and 35% longer than Bayes-adaptive Reinforcement Learning. While both SL and SI showed equal convergence performance, SL reached this convergence 40% faster than SI. Additionally, SL showed robust out-performance of other algorithms in altered environment configurations. Our results show that incorporating active learning into multi-step planning materially improves decision making under radical uncertainty, and reinforces the broader utility of Active Inference for modeling biologically relevant behavior.