Evolving Hierarchical Memory-Prediction Machines in Multi-Task Reinforcement Learning
This addresses the challenge of building adaptable AI agents for varied control tasks, though it is incremental as it builds on existing genetic programming and multi-task learning methods.
The paper tackled the problem of creating generalized agents for multi-task reinforcement learning in dynamic environments without task-identification inputs, using genetic programming to evolve agents that are competitive with task-specific ones across six control environments, including OpenAI's Classic Control suite.
A fundamental aspect of behaviour is the ability to encode salient features of experience in memory and use these memories, in combination with current sensory information, to predict the best action for each situation such that long-term objectives are maximized. The world is highly dynamic, and behavioural agents must generalize across a variety of environments and objectives over time. This scenario can be modeled as a partially-observable multi-task reinforcement learning problem. We use genetic programming to evolve highly-generalized agents capable of operating in six unique environments from the control literature, including OpenAI's entire Classic Control suite. This requires the agent to support discrete and continuous actions simultaneously. No task-identification sensor inputs are provided, thus agents must identify tasks from the dynamics of state variables alone and define control policies for each task. We show that emergent hierarchical structure in the evolving programs leads to multi-task agents that succeed by performing a temporal decomposition and encoding of the problem environments in memory. The resulting agents are competitive with task-specific agents in all six environments. Furthermore, the hierarchical structure of programs allows for dynamic run-time complexity, which results in relatively efficient operation.