Credit Assignment: Challenges and Opportunities in Developing Human-like AI Agents
This research addresses the challenge of developing more human-like AI agents for applications in cognitive science and AI systems, though it is incremental as it builds on existing theories and models.
The study tackled the problem of temporal credit assignment in human-like AI agents by testing different credit assignment mechanisms in a goal-seeking navigation task, finding that an Instance-Based Learning model with equal credit assignment matched human performance better than other models, with IBL-TD and Q-learning initially underperforming but eventually outperforming humans.
Temporal credit assignment is crucial for learning and skill development in natural and artificial intelligence. While computational methods like the TD approach in reinforcement learning have been proposed, it's unclear if they accurately represent how humans handle feedback delays. Cognitive models intend to represent the mental steps by which humans solve problems and perform a number of tasks, but limited research in cognitive science has addressed the credit assignment problem in humans and cognitive models. Our research uses a cognitive model based on a theory of decisions from experience, Instance-Based Learning Theory (IBLT), to test different credit assignment mechanisms in a goal-seeking navigation task with varying levels of decision complexity. Instance-Based Learning (IBL) models simulate the process of making sequential choices with different credit assignment mechanisms, including a new IBL-TD model that combines the IBL decision mechanism with the TD approach. We found that (1) An IBL model that gives equal credit assignment to all decisions is able to match human performance better than other models, including IBL-TD and Q-learning; (2) IBL-TD and Q-learning models underperform compared to humans initially, but eventually, they outperform humans; (3) humans are influenced by decision complexity, while models are not. Our study provides insights into the challenges of capturing human behavior and the potential opportunities to use these models in future AI systems to support human activities.