Cortex and subcortex play distinct roles over learning when cortical memory is limited
For computational neuroscience and AI, this work offers a theoretical account of how limited cortical memory may drive distinct roles for cortical and subcortical systems during learning.
The paper extends theoretical frameworks of model-based and model-free learning by constraining memory resources of the model-based module, showing that when rewarded states change often, it is advantageous for the model-based module to focus on learning general environmental structure rather than exploiting current rewards. This provides a theoretical foundation for a functional dissociation between cortex (general structure learning) and subcortex (reward-based learning).
It has been proposed that the brain integrates flexible, computationally expensive cortical processing with simpler, lower-cost subcortical mechanisms to achieve resource-efficient performance greater than that of either system alone. Despite the allure of this perspective, satisfying theoretical frameworks that explore this hypothesis are still limited. We extend existing frameworks in which a model-based module and model-free module learn in tandem by explicitly constraining the memory resources of the model-based module, and investigate the impact of this constraint in a simple decision-making setting. Memory constraints naturally give rise to strategies for allocating memory resources. We evaluate the performance of different strategies in different situations and demonstrate that when the rewarded states change often, it can be advantageous for the model-based module to focus its memory resources not on exploiting the current reward, but on capturing general structure of the environment. This work provides a theoretical foundation for a functional dissociation between cortical and subcortical systems during learning: the cortex supports general structure learning, while subcortical circuits specialize in reward-based learning. We further detail how these hypotheses can be tested on experimental data.