Evolution of Rewards for Food and Motor Action by Simulating Birth and Death
This work addresses the underexplored problem of how reward systems evolve, using a simulation framework to investigate environmental effects, but it is incremental as it builds on existing evolutionary and reinforcement learning models.
The study tackled the evolution of biologically plausible reward functions by simulating agents that maintain energy to live and reproduce, finding that positive rewards for food acquisition and negative rewards for motor action can evolve from random initialization, with motor action rewards diverging into positive and negative modes, and rewards for less important foods being unstable in poor environments.
The reward system is one of the fundamental drivers of animal behaviors and is critical for survival and reproduction. Despite its importance, the problem of how the reward system has evolved is underexplored. In this paper, we try to replicate the evolution of biologically plausible reward functions and investigate how environmental conditions affect evolved rewards' shape. For this purpose, we developed a population-based decentralized evolutionary simulation framework, where agents maintain their energy level to live longer and produce more children. Each agent inherits its reward function from its parent subject to mutation and learns to get rewards via reinforcement learning throughout its lifetime. Our results show that biologically reasonable positive rewards for food acquisition and negative rewards for motor action can evolve from randomly initialized ones. However, we also find that the rewards for motor action diverge into two modes: largely positive and slightly negative. The emergence of positive motor action rewards is surprising because it can make agents too active and inefficient in foraging. In environments with poor and poisonous foods, the evolution of rewards for less important foods tends to be unstable, while rewards for normal foods are still stable. These results demonstrate the usefulness of our simulation environment and energy-dependent birth and death model for further studies of the origin of reward systems.