Combining imitation and deep reinforcement learning to accomplish human-level performance on a virtual foraging task
This work addresses the challenge of developing bio-inspired foraging policies for virtual environments, but it is incremental as it builds on existing IL and RL methods without introducing major innovations.
The authors tackled the problem of achieving human-level performance in a virtual foraging task by combining imitation learning (IL) from human data with reinforcement learning (RL), specifically using Proximal Policy Optimization (PPO). They found that this combined approach matched human results, with performance strongly dependent on integrating allocentric and egocentric environmental representations.
We develop a simple framework to learn bio-inspired foraging policies using human data. We conduct an experiment where humans are virtually immersed in an open field foraging environment and are trained to collect the highest amount of rewards. A Markov Decision Process (MDP) framework is introduced to model the human decision dynamics. Then, Imitation Learning (IL) based on maximum likelihood estimation is used to train Neural Networks (NN) that map human decisions to observed states. The results show that passive imitation substantially underperforms humans. We further refine the human-inspired policies via Reinforcement Learning (RL) using the on-policy Proximal Policy Optimization (PPO) algorithm which shows better stability than other algorithms and can steadily improve the policies pretrained with IL. We show that the combination of IL and RL can match human results and that good performance strongly depends on combining the allocentric information with an egocentric representation of the environment.